[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:32 dist-test-slave-1fdt krb5kdc[24414](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:32 dist-test-slave-1fdt krb5kdc[24414](info): set up 2 sockets
Sep 26 05:41:32 dist-test-slave-1fdt krb5kdc[24414](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 05:41:33.927287 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.733s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260926 05:41:34.083842 24407 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.156s user 0.002s sys 0.000s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:34.184710 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33021
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:33021
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:34.265909 24427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:34.266117 24427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:34.266140 24427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:34.267316 24427 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:34.267359 24427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:34.267373 24427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:34.267414 24427 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:34.267444 24427 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:34.268906 24427 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:33021
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33021
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:34.269177 24427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:34.269418 24427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:34.274634 24433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.274752 24432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.274646 24435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:34.274657 24427 server_base.cc:1034] running on GCE node
I20260926 05:41:34.275204 24427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:34.275563 24427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:34.276733 24427 hybrid_clock.cc:648] HybridClock initialized: now 1790401294276717 us; error 46 us; skew 500 ppm
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:34.284607 24427 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:34.285094 24427 webserver.cc:460] Webserver started at http://127.23.213.254:45329/ using document root <none> and password file <none>
I20260926 05:41:34.285362 24427 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:34.285437 24427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:34.285588 24427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:34.286794 24427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "2c1122bbfbb24da2be583975a8a0a7fa"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "9e42a8abb2415e476b96024600e31a5f"
server_key_iv: "6c2772c538bf9c3dbfc501cf49e23713"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.287176 24427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "2c1122bbfbb24da2be583975a8a0a7fa"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "9e42a8abb2415e476b96024600e31a5f"
server_key_iv: "6c2772c538bf9c3dbfc501cf49e23713"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.288545 24427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:34.289338 24442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.289505 24427 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:34.289600 24427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "2c1122bbfbb24da2be583975a8a0a7fa"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "9e42a8abb2415e476b96024600e31a5f"
server_key_iv: "6c2772c538bf9c3dbfc501cf49e23713"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.289712 24427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:34.300577 24427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:34.302469 24427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:34.302634 24427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:34.306809 24427 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:33021
I20260926 05:41:34.306855 24494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:33021 every 8 connection(s)
I20260926 05:41:34.307236 24427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:34.308159 24495 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:34.309330 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 24427
I20260926 05:41:34.309409 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:34.310089 24407 external_mini_cluster.cc:1428] Setting key b4688281986b746d41bc286c2ac93075
I20260926 05:41:34.310634 24495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa: Bootstrap starting.
I20260926 05:41:34.311393 24495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:34.311705 24495 log.cc:824] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:34.312405 24495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa: No bootstrap required, opened a new log
I20260926 05:41:34.314324 24495 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33021 } }
I20260926 05:41:34.314494 24495 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:34.314529 24495 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c1122bbfbb24da2be583975a8a0a7fa, State: Initialized, Role: FOLLOWER
I20260926 05:41:34.314687 24495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33021 } }
I20260926 05:41:34.314754 24495 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:34.314774 24495 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:34.314841 24495 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:34.315644 24495 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33021 } }
I20260926 05:41:34.315735 24495 leader_election.cc:302] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c1122bbfbb24da2be583975a8a0a7fa; no voters:
I20260926 05:41:34.315953 24495 leader_election.cc:288] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:34.315976 24500 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:34.316082 24500 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [term 1 LEADER]: Becoming Leader. State: Replica: 2c1122bbfbb24da2be583975a8a0a7fa, State: Running, Role: LEADER
I20260926 05:41:34.316213 24495 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:34.316174 24500 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33021 } }
I20260926 05:41:34.316542 24502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c1122bbfbb24da2be583975a8a0a7fa. Latest consensus state: current_term: 1 leader_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33021 } } }
I20260926 05:41:34.316639 24502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:34.316818 24501 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c1122bbfbb24da2be583975a8a0a7fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33021 } } }
I20260926 05:41:34.316879 24501 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa [sys.catalog]: This master's current role is: LEADER
W20260926 05:41:34.318053 24516 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:41:34.318110 24516 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:41:34.318154 24506 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:34.318320 24506 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:34.320644 24506 catalog_manager.cc:1347] Generated new cluster ID: c7ba8256bbb74652bde96b2ad91e2911
I20260926 05:41:34.320714 24506 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:34.327919 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.310972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38866 (local address 127.23.213.254:33021)
0926 05:41:34.311174 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:34.311183 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:34.311404 (+ 221us) server_negotiation.cc:407] Connection header received
0926 05:41:34.314281 (+ 2877us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:34.314289 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:34.314352 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:34.314497 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:34.316297 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.317726 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.319195 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.319704 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.319827 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.323847 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:34.323851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:34.323851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:34.323862 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:34.325862 (+ 2000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.326385 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.326388 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.326388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.326500 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.326817 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.326818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.326818 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.327096 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:34.327239 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:34.327559 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 05:41:34.327702 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":87,"threads_started":1}
I20260926 05:41:34.331151 24506 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:34.331866 24506 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:34.338034 24506 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c1122bbfbb24da2be583975a8a0a7fa: Generated new TSK 0
I20260926 05:41:34.338217 24506 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:34.366655 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33021
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:34.448001 24521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:34.448248 24521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:34.448292 24521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:34.449431 24521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:34.449497 24521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:34.449542 24521 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:34.450987 24521 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:34.451254 24521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:34.451488 24521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:34.456437 24526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.456508 24527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.456442 24529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:34.456724 24521 server_base.cc:1034] running on GCE node
I20260926 05:41:34.456986 24521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:34.457235 24521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:34.458393 24521 hybrid_clock.cc:648] HybridClock initialized: now 1790401294458373 us; error 35 us; skew 500 ppm
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:34.460903 24521 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:34.461364 24521 webserver.cc:460] Webserver started at http://127.23.213.193:45929/ using document root <none> and password file <none>
I20260926 05:41:34.461539 24521 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:34.461576 24521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:34.461679 24521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:34.462749 24521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "d576d96a803142639bbb9161af975918"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "35cb07531b0cf25ee002b263db47a40e"
server_key_iv: "40f19a1641a42e744bdf47907cfb0449"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.463047 24521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "d576d96a803142639bbb9161af975918"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "35cb07531b0cf25ee002b263db47a40e"
server_key_iv: "40f19a1641a42e744bdf47907cfb0449"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.464304 24521 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:34.464964 24536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.465116 24521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:34.465202 24521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "d576d96a803142639bbb9161af975918"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "35cb07531b0cf25ee002b263db47a40e"
server_key_iv: "40f19a1641a42e744bdf47907cfb0449"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.465256 24521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:34.477283 24521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:34.479128 24521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:34.479254 24521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:34.479435 24521 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:34.479737 24521 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:34.479770 24521 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.479811 24521 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:34.479838 24521 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.485492 24521 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:41239
I20260926 05:41:34.485579 24649 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:41239 every 8 connection(s)
I20260926 05:41:34.485914 24521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:34.491571 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 24521
I20260926 05:41:34.491680 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:34.491851 24407 external_mini_cluster.cc:1428] Setting key 1fe12d793126d874ca289849f16d8e24
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:34.495720 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.486602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:49677 (local address 127.23.213.254:33021)
0926 05:41:34.486682 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:34.486685 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:34.486884 (+ 199us) server_negotiation.cc:407] Connection header received
0926 05:41:34.487486 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:34.487487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:34.487504 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:34.487550 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:34.488521 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.489033 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.490120 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.490349 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.490446 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.492774 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:34.492775 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:34.492776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:34.492787 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:34.494267 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.494702 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.494704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.494704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.494765 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.495023 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.495024 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.495024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.495137 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:34.495242 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:34.495537 (+ 295us) server_negotiation.cc:299] Negotiation successful
0926 05:41:34.495631 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:41:34.495915 24652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.486564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:33021 (local address 127.23.213.193:49677)
0926 05:41:34.486771 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:34.486804 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:34.487367 (+ 563us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:34.487636 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:34.487640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:34.488223 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:34.488411 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:34.488420 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.489132 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.489133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:34.490023 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:34.490025 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.490457 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.490458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:34.490579 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.491029 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:34.491037 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:34.492673 (+ 1636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:34.494373 (+ 1700us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:34.494380 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:34.494382 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:34.494607 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:34.494836 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:34.494836 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:34.494837 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:34.494947 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:34.495235 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:34.495241 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:34.495381 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 05:41:34.495503 (+ 122us) client_negotiation.cc:240] Negotiation successful
0926 05:41:34.495661 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":96,"threads_started":1}
I20260926 05:41:34.496433 24650 heartbeater.cc:347] Connected to a master server at 127.23.213.254:33021
I20260926 05:41:34.496536 24650 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:34.496713 24650 heartbeater.cc:511] Master 127.23.213.254:33021 requested a full tablet report, sending...
I20260926 05:41:34.497365 24459 ts_manager.cc:194] Registered new tserver with Master: d576d96a803142639bbb9161af975918 (127.23.213.193:41239)
I20260926 05:41:34.498342 24459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:49677
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:34.516877 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33021
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:34.598300 24655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:34.598498 24655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:34.598523 24655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:34.599671 24655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:34.599711 24655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:34.599768 24655 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:34.601241 24655 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:34.601467 24655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:34.601691 24655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:34.606462 24661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.606478 24660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.606444 24663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:34.606658 24655 server_base.cc:1034] running on GCE node
I20260926 05:41:34.606863 24655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:34.607084 24655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:34.608237 24655 hybrid_clock.cc:648] HybridClock initialized: now 1790401294608222 us; error 39 us; skew 500 ppm
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:34.610687 24655 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:34.611193 24655 webserver.cc:460] Webserver started at http://127.23.213.194:40861/ using document root <none> and password file <none>
I20260926 05:41:34.611410 24655 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:34.611472 24655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:34.611604 24655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:34.612600 24655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "6d429e7fd7324a2f8042eb00169f7146"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "4c1e6880739bb82e43bbb40b4ff2bffc"
server_key_iv: "9c5997797466fd40faf6a3d782157590"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.612982 24655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "6d429e7fd7324a2f8042eb00169f7146"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "4c1e6880739bb82e43bbb40b4ff2bffc"
server_key_iv: "9c5997797466fd40faf6a3d782157590"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.614337 24655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:34.615049 24670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.615208 24655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:34.615305 24655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "6d429e7fd7324a2f8042eb00169f7146"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "4c1e6880739bb82e43bbb40b4ff2bffc"
server_key_iv: "9c5997797466fd40faf6a3d782157590"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.615403 24655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:34.629570 24655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:34.631546 24655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:34.631712 24655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:34.631939 24655 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:34.632254 24655 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:34.632306 24655 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.632345 24655 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:34.632375 24655 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.638077 24655 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:46103
I20260926 05:41:34.638120 24783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:46103 every 8 connection(s)
I20260926 05:41:34.638441 24655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:41:34.641588 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 24655
I20260926 05:41:34.641655 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:34.641846 24407 external_mini_cluster.cc:1428] Setting key 663442aa59b1920469919e2165d895d6
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:34.648633 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.639157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:37905 (local address 127.23.213.254:33021)
0926 05:41:34.639221 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:34.639223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:34.639448 (+ 225us) server_negotiation.cc:407] Connection header received
0926 05:41:34.639917 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:34.639917 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:34.639926 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:34.639979 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:34.640900 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.641386 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.642717 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.643034 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.643113 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.645532 (+ 2419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:34.645533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:34.645534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:34.645542 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:34.647271 (+ 1729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.647654 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.647655 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.647655 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.647729 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.647962 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.647963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.647963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.648112 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:34.648187 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:34.648423 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 05:41:34.648518 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:41:34.648644 24786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.639138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:33021 (local address 127.23.213.194:37905)
0926 05:41:34.639317 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:34.639359 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:34.639836 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:34.640043 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:34.640046 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:34.640620 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:34.640822 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:34.640827 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.641470 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.641470 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:34.642639 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:34.642642 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.643121 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.643122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:34.643226 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.643751 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:34.643784 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:34.645436 (+ 1652us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:34.647352 (+ 1916us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:34.647358 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:34.647359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:34.647575 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:34.647799 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:34.647799 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:34.647800 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:34.647893 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:34.648178 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:34.648182 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:34.648297 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 05:41:34.648375 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 05:41:34.648490 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260926 05:41:34.649325 24784 heartbeater.cc:347] Connected to a master server at 127.23.213.254:33021
I20260926 05:41:34.649434 24784 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:34.649606 24784 heartbeater.cc:511] Master 127.23.213.254:33021 requested a full tablet report, sending...
I20260926 05:41:34.650028 24459 ts_manager.cc:194] Registered new tserver with Master: 6d429e7fd7324a2f8042eb00169f7146 (127.23.213.194:46103)
I20260926 05:41:34.650413 24459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:37905
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:34.668147 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33021
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36891
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:34.747488 24789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:34.747689 24789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:34.747712 24789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:34.748868 24789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:34.748910 24789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:34.748937 24789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:34.750389 24789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36891
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33021
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:34.750625 24789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:34.750854 24789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:34.756044 24795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.756234 24794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:34.756079 24797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:34.756276 24789 server_base.cc:1034] running on GCE node
I20260926 05:41:34.756464 24789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:34.756673 24789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:34.757826 24789 hybrid_clock.cc:648] HybridClock initialized: now 1790401294757812 us; error 27 us; skew 500 ppm
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:34.760288 24789 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:34.760759 24789 webserver.cc:460] Webserver started at http://127.23.213.195:38571/ using document root <none> and password file <none>
I20260926 05:41:34.760943 24789 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:34.760983 24789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:34.761080 24789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:34.762169 24789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "f52cd0bc526f4579beeb8ec4832a7634"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "298e9351bfcff5c170dc6c3bdc3513a4"
server_key_iv: "c3db14d621437ec8c75a1073a16a3701"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.762533 24789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "f52cd0bc526f4579beeb8ec4832a7634"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "298e9351bfcff5c170dc6c3bdc3513a4"
server_key_iv: "c3db14d621437ec8c75a1073a16a3701"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.763844 24789 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:34.764533 24804 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.764724 24789 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:34.764824 24789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "f52cd0bc526f4579beeb8ec4832a7634"
format_stamp: "Formatted at 2026-09-26 05:41:34 on dist-test-slave-1fdt"
server_key: "298e9351bfcff5c170dc6c3bdc3513a4"
server_key_iv: "c3db14d621437ec8c75a1073a16a3701"
server_key_version: "encryptionkey@0"
I20260926 05:41:34.764899 24789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:34.780673 24789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:34.782823 24789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:34.782990 24789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:34.783250 24789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:34.783581 24789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:34.783636 24789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.783679 24789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:34.783744 24789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:34.789963 24789 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:37353
I20260926 05:41:34.789991 24917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:37353 every 8 connection(s)
I20260926 05:41:34.790349 24789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:34.792912 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 24789
I20260926 05:41:34.792989 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:34.793219 24407 external_mini_cluster.cc:1428] Setting key 03a4b97b95e5dfeb5af64611f61f398e
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:34.800549 24497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.791184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:47967 (local address 127.23.213.254:33021)
0926 05:41:34.791243 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:34.791245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:34.791423 (+ 178us) server_negotiation.cc:407] Connection header received
0926 05:41:34.791925 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:34.791926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:34.791937 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:34.791989 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:34.793168 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.793933 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.795187 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.795395 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.795480 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.797576 (+ 2096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:34.797578 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:34.797578 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:34.797586 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:34.799060 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.799495 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.799496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.799497 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.799558 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.799811 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.799812 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.799812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.799941 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:34.800026 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:34.800334 (+ 308us) server_negotiation.cc:299] Negotiation successful
0926 05:41:34.800414 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:41:34.800733 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.791168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:33021 (local address 127.23.213.195:47967)
0926 05:41:34.791320 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:34.791357 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:34.791838 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:34.792087 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:34.792090 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:34.792718 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:34.793046 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:34.793052 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.794174 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.794175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:34.795082 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:34.795084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.795489 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.795490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:34.795623 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.796094 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:34.796103 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:34.797472 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:34.799166 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:34.799172 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:34.799175 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:34.799406 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:34.799634 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:34.799634 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:34.799635 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:34.799744 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:34.800015 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:34.800022 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:34.800179 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 05:41:34.800287 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 05:41:34.800459 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260926 05:41:34.801280 24918 heartbeater.cc:347] Connected to a master server at 127.23.213.254:33021
I20260926 05:41:34.801385 24918 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:34.801530 24918 heartbeater.cc:511] Master 127.23.213.254:33021 requested a full tablet report, sending...
I20260926 05:41:34.801899 24459 ts_manager.cc:194] Registered new tserver with Master: f52cd0bc526f4579beeb8ec4832a7634 (127.23.213.195:37353)
I20260926 05:41:34.802299 24459 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:47967
I20260926 05:41:34.805325 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:34.828388 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.820768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53166 (local address 127.23.213.193:41239)
0926 05:41:34.820935 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:34.820947 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:34.820959 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:41:34.820987 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:34.820989 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:34.821003 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:34.821079 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:34.821637 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.822412 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.823123 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.823451 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.823505 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.825715 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:34.825725 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:34.825725 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:34.825732 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:34.827373 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.827674 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.827676 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.827676 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.827728 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.827887 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.827888 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.827888 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.828021 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:34.828122 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:34.828247 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 05:41:34.828308 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260926 05:41:34.828814 24564 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53166
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24414](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401294, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:34.850956 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:34.844084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53174 (local address 127.23.213.193:41239)
0926 05:41:34.844143 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:34.844145 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:34.844250 (+ 105us) server_negotiation.cc:407] Connection header received
0926 05:41:34.844298 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:34.844299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:34.844311 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:34.844370 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:34.844912 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.845385 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.846167 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:34.846362 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:34.846417 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:34.848575 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:34.848576 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:34.848576 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:34.848584 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:34.850038 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.850333 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.850334 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.850335 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.850383 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:34.850570 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:34.850570 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:34.850570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:34.850667 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:34.850755 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:34.850828 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 05:41:34.850889 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:41:34.851541 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 24521
I20260926 05:41:34.856333 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 24655
I20260926 05:41:34.861866 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 24789
I20260926 05:41:34.866197 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 24427
2026-09-26T05:41:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2697 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24932](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24932](info): set up 2 sockets
Sep 26 05:41:34 dist-test-slave-1fdt krb5kdc[24932](info): commencing operation
krb5kdc: starting...
W20260926 05:41:35.905635 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:35 dist-test-slave-1fdt krb5kdc[24932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:36.018846 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38685
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38685
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:36.097470 24945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:36.097669 24945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:36.097692 24945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:36.098862 24945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:36.098903 24945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:36.098918 24945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:36.098930 24945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:36.098942 24945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:36.100301 24945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38685
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38685
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24945
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:36.100481 24945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:36.100641 24945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:36.105386 24951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:36.105361 24953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:36.105418 24950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:36.105479 24945 server_base.cc:1034] running on GCE node
I20260926 05:41:36.105926 24945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:36.106237 24945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:36.107386 24945 hybrid_clock.cc:648] HybridClock initialized: now 1790401296107376 us; error 32 us; skew 500 ppm
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:36.110018 24945 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:36.110387 24945 webserver.cc:460] Webserver started at http://127.23.213.254:33955/ using document root <none> and password file <none>
I20260926 05:41:36.110636 24945 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:36.110713 24945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:36.110894 24945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:36.111977 24945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "a21659b08b2e4fb4b40cf3075d340a5e"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "8c8ca3851956903a0c208b6965fb6b09"
server_key_iv: "7d94841ec680d3dae444122d1c185cbe"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.112375 24945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "a21659b08b2e4fb4b40cf3075d340a5e"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "8c8ca3851956903a0c208b6965fb6b09"
server_key_iv: "7d94841ec680d3dae444122d1c185cbe"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.113821 24945 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 05:41:36.114558 24960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.114730 24945 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:36.114823 24945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "a21659b08b2e4fb4b40cf3075d340a5e"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "8c8ca3851956903a0c208b6965fb6b09"
server_key_iv: "7d94841ec680d3dae444122d1c185cbe"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.114910 24945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:36.124511 24945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:36.126839 24945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:36.127004 24945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:36.130860 24945 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38685
I20260926 05:41:36.130895 25012 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38685 every 8 connection(s)
I20260926 05:41:36.131264 24945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:36.132122 25013 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:36.133296 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 24945
I20260926 05:41:36.133363 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:36.133522 24407 external_mini_cluster.cc:1428] Setting key a6a689af337cba10260aa1434fd14123
I20260926 05:41:36.134698 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e: Bootstrap starting.
I20260926 05:41:36.135444 25013 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:36.135751 25013 log.cc:824] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:36.136415 25013 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e: No bootstrap required, opened a new log
I20260926 05:41:36.138890 25013 raft_consensus.cc:348] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38685 } }
I20260926 05:41:36.139065 25013 raft_consensus.cc:374] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:36.139103 25013 raft_consensus.cc:729] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a21659b08b2e4fb4b40cf3075d340a5e, State: Initialized, Role: FOLLOWER
I20260926 05:41:36.139262 25013 consensus_queue.cc:260] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38685 } }
I20260926 05:41:36.139353 25013 raft_consensus.cc:388] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:36.139391 25013 raft_consensus.cc:482] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:36.139444 25013 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:36.140064 25013 raft_consensus.cc:504] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38685 } }
I20260926 05:41:36.140190 25013 leader_election.cc:302] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a21659b08b2e4fb4b40cf3075d340a5e; no voters:
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:36.140372 25013 leader_election.cc:288] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:36.140420 25018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:36.140590 25018 raft_consensus.cc:686] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [term 1 LEADER]: Becoming Leader. State: Replica: a21659b08b2e4fb4b40cf3075d340a5e, State: Running, Role: LEADER
I20260926 05:41:36.140621 25013 sys_catalog.cc:564] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:36.140684 25018 consensus_queue.cc:237] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38685 } }
I20260926 05:41:36.140967 25018 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a21659b08b2e4fb4b40cf3075d340a5e. Latest consensus state: current_term: 1 leader_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38685 } } }
I20260926 05:41:36.141057 25018 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:36.140978 25020 sys_catalog.cc:455] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a21659b08b2e4fb4b40cf3075d340a5e" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38685 } } }
I20260926 05:41:36.141857 25020 sys_catalog.cc:458] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e [sys.catalog]: This master's current role is: LEADER
W20260926 05:41:36.142407 25034 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:41:36.142599 25034 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:41:36.142753 25027 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:36.142930 25027 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:36.144418 25027 catalog_manager.cc:1347] Generated new cluster ID: 670416036c8b467c99387dbe8ff2d701
I20260926 05:41:36.144480 25027 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:36.144649 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.134071 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.23.213.254:38685)
0926 05:41:36.134232 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.134248 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.134273 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:41:36.134672 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.134678 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.134739 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.134907 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.135638 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.136598 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.137639 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.138095 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.138171 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.140840 (+ 2669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.140845 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.140846 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.140857 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.142911 (+ 2054us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.143284 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.143287 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.143287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.143401 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.143620 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.143621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.143621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.143948 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.144116 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.144229 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.144370 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260926 05:41:36.154609 25027 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:36.155117 25027 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:36.159327 25027 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a21659b08b2e4fb4b40cf3075d340a5e: Generated new TSK 0
I20260926 05:41:36.159498 25027 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:36.188755 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38685
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:36.290333 25039 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:36.290576 25039 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:36.290618 25039 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:36.291792 25039 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:36.291859 25039 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:36.291908 25039 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:36.293423 25039 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38685
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25039
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:36.293707 25039 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:36.293977 25039 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:36.299886 25044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:36.299903 25047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:36.300009 25045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:36.300280 25039 server_base.cc:1034] running on GCE node
I20260926 05:41:36.300482 25039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:36.300712 25039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:36.301877 25039 hybrid_clock.cc:648] HybridClock initialized: now 1790401296301849 us; error 40 us; skew 500 ppm
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:36.304430 25039 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:36.304843 25039 webserver.cc:460] Webserver started at http://127.23.213.193:41371/ using document root <none> and password file <none>
I20260926 05:41:36.305073 25039 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:36.305140 25039 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:36.305296 25039 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:36.306401 25039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "e1e877247a7a4d4aafa02de9e705e5ba"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "57b1b4e450ea2fadb5931d5249272840"
server_key_iv: "e151d227311f4eb3ab3eb13bd610d604"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.306754 25039 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "e1e877247a7a4d4aafa02de9e705e5ba"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "57b1b4e450ea2fadb5931d5249272840"
server_key_iv: "e151d227311f4eb3ab3eb13bd610d604"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.308013 25039 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:36.308697 25054 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.308868 25039 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:36.308969 25039 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "e1e877247a7a4d4aafa02de9e705e5ba"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "57b1b4e450ea2fadb5931d5249272840"
server_key_iv: "e151d227311f4eb3ab3eb13bd610d604"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.309042 25039 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:36.333901 25039 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:36.335829 25039 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:36.335997 25039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:36.336243 25039 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:36.336573 25039 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:36.336632 25039 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.336674 25039 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:36.336707 25039 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.342890 25039 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:46123
I20260926 05:41:36.342935 25167 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:46123 every 8 connection(s)
I20260926 05:41:36.343312 25039 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:36.344439 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 25039
I20260926 05:41:36.344544 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:36.344712 24407 external_mini_cluster.cc:1428] Setting key 7d9b9ece7ac005879fb93778630d026a
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
I20260926 05:41:36.353497 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.344092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:48101 (local address 127.23.213.254:38685)
0926 05:41:36.344148 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.344150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.344362 (+ 212us) server_negotiation.cc:407] Connection header received
0926 05:41:36.345434 (+ 1072us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.345435 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.345447 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.345497 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.346202 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.346885 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.347926 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.348290 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.348366 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.350590 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.350591 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.350592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.350600 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.352199 (+ 1599us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.352573 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.352574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.352574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.352635 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.352912 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.352913 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.352913 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.353061 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.353126 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.353365 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.353433 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:41:36.353662 25170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.344079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38685 (local address 127.23.213.193:48101)
0926 05:41:36.344247 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.344282 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.345335 (+ 1053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.345567 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.345571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.345907 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:36.346097 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.346102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.346986 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.346987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.347825 (+ 838us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.347828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.348383 (+ 555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.348384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.348486 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.349091 (+ 605us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:36.349099 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:36.350493 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:36.352278 (+ 1785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:36.352284 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:36.352286 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:36.352500 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:36.352712 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:36.352713 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:36.352713 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:36.352861 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:36.353130 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:36.353134 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:36.353250 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.353331 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.353445 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
I20260926 05:41:36.354213 25168 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38685
I20260926 05:41:36.354328 25168 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:36.354497 25168 heartbeater.cc:511] Master 127.23.213.254:38685 requested a full tablet report, sending...
I20260926 05:41:36.355000 24977 ts_manager.cc:194] Registered new tserver with Master: e1e877247a7a4d4aafa02de9e705e5ba (127.23.213.193:46123)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
I20260926 05:41:36.355710 24977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:48101
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:36.368765 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38685
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:36.448880 25173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:36.449079 25173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:36.449102 25173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:36.450227 25173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:36.450266 25173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:36.450292 25173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:36.451678 25173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38685
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25173
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:36.451849 25173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:36.452013 25173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:36.456888 25178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:36.456935 25173 server_base.cc:1034] running on GCE node
W20260926 05:41:36.457067 25181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:36.457098 25179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:36.457366 25173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:36.457556 25173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:36.458716 25173 hybrid_clock.cc:648] HybridClock initialized: now 1790401296458701 us; error 27 us; skew 500 ppm
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:36.461094 25173 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:36.461550 25173 webserver.cc:460] Webserver started at http://127.23.213.194:43715/ using document root <none> and password file <none>
I20260926 05:41:36.461835 25173 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:36.461903 25173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:36.462037 25173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:36.463050 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "d98adac83fd941dba11b15f24a79827f"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "f5401e514afc02c5283aecb243124f34"
server_key_iv: "c17a8907b519f23cadd67f04929a5b16"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.463400 25173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "d98adac83fd941dba11b15f24a79827f"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "f5401e514afc02c5283aecb243124f34"
server_key_iv: "c17a8907b519f23cadd67f04929a5b16"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.464705 25173 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:36.465363 25188 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.465523 25173 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:36.465628 25173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "d98adac83fd941dba11b15f24a79827f"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "f5401e514afc02c5283aecb243124f34"
server_key_iv: "c17a8907b519f23cadd67f04929a5b16"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.465721 25173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:36.486661 25173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:36.488379 25173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:36.488538 25173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:36.488775 25173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:36.489073 25173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:36.489128 25173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.489179 25173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:36.489210 25173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.495044 25173 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:39577
I20260926 05:41:36.495095 25301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:39577 every 8 connection(s)
I20260926 05:41:36.495402 25173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:36.503355 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 25173
I20260926 05:41:36.503444 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:36.503638 24407 external_mini_cluster.cc:1428] Setting key df6a347b60d628ef0210c6986938651e
I20260926 05:41:36.505479 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.496136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:41631 (local address 127.23.213.254:38685)
0926 05:41:36.496194 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.496196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.496405 (+ 209us) server_negotiation.cc:407] Connection header received
0926 05:41:36.497390 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.497391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.497404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.497457 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.498391 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.499018 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.500076 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.500285 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.500371 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.502581 (+ 2210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.502582 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.502582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.502591 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.504084 (+ 1493us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.504552 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.504553 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.504553 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.504608 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.504881 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.504882 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.504882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.504979 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.505028 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.505352 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.505417 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:41:36.505810 25304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.496110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38685 (local address 127.23.213.194:41631)
0926 05:41:36.496283 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.496325 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.497267 (+ 942us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.497526 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.497530 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.497942 (+ 412us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:36.498284 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.498290 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.499111 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.499112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.499951 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.499953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.500383 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.500384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.500526 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.501067 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:36.501075 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:36.502481 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:36.504172 (+ 1691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:36.504187 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:36.504189 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:36.504470 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:36.504670 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:36.504671 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:36.504671 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:36.504816 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:36.505042 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:36.505047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:36.505221 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.505332 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.505509 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260926 05:41:36.506259 25302 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38685
I20260926 05:41:36.506337 25302 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:36.506528 25302 heartbeater.cc:511] Master 127.23.213.254:38685 requested a full tablet report, sending...
I20260926 05:41:36.506896 24977 ts_manager.cc:194] Registered new tserver with Master: d98adac83fd941dba11b15f24a79827f (127.23.213.194:39577)
I20260926 05:41:36.507436 24977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:41631
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:36.528755 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38685
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:35833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:36.611199 25307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:36.611392 25307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:36.611414 25307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:36.612627 25307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:36.612666 25307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:36.612722 25307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:36.614301 25307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38685
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:36.614554 25307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:36.614773 25307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:36.619817 25315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:36.619814 25313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:36.619886 25312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:36.620265 25307 server_base.cc:1034] running on GCE node
I20260926 05:41:36.620491 25307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:36.620738 25307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:36.621896 25307 hybrid_clock.cc:648] HybridClock initialized: now 1790401296621887 us; error 39 us; skew 500 ppm
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:36.624344 25307 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:36.624722 25307 webserver.cc:460] Webserver started at http://127.23.213.195:36823/ using document root <none> and password file <none>
I20260926 05:41:36.624909 25307 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:36.624948 25307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:36.625036 25307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:36.626080 25307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "e8263434f9e5d1a6e3f3dcc88e91aa71"
server_key_iv: "36677cd54b3ec444827cbfb1cf5402fe"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.626379 25307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "e8263434f9e5d1a6e3f3dcc88e91aa71"
server_key_iv: "36677cd54b3ec444827cbfb1cf5402fe"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.627614 25307 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:36.628288 25322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.628453 25307 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:36.628533 25307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6"
format_stamp: "Formatted at 2026-09-26 05:41:36 on dist-test-slave-1fdt"
server_key: "e8263434f9e5d1a6e3f3dcc88e91aa71"
server_key_iv: "36677cd54b3ec444827cbfb1cf5402fe"
server_key_version: "encryptionkey@0"
I20260926 05:41:36.628605 25307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:36.640626 25307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:36.642976 25307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:36.643136 25307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:36.643365 25307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:36.643662 25307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:36.643718 25307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.643757 25307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:36.643784 25307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:36.649147 25307 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:38791
I20260926 05:41:36.649252 25435 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:38791 every 8 connection(s)
I20260926 05:41:36.649541 25307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:36.653519 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 25307
I20260926 05:41:36.653610 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:36.653815 24407 external_mini_cluster.cc:1428] Setting key c20c1e1ed3cffb8cc9d9f6e2a4bb805b
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:36.658880 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.650330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:56781 (local address 127.23.213.254:38685)
0926 05:41:36.650410 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.650412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.650571 (+ 159us) server_negotiation.cc:407] Connection header received
0926 05:41:36.651445 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.651446 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.651456 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.651502 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.652087 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.652572 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.653791 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.654095 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.654179 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.656335 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.656337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.656337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.656343 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.657721 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.658126 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.658127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.658127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.658180 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.658382 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.658382 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.658383 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.658478 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.658527 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.658762 (+ 235us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.658818 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:41:36.659042 25438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.650297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38685 (local address 127.23.213.195:56781)
0926 05:41:36.650448 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.650484 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.651361 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.651565 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.651568 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.651811 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:36.652018 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.652022 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.652648 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.652649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.653618 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.653621 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.654226 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.654227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.654378 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.654922 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:36.654930 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:36.656253 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:36.657849 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:36.657854 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:36.657856 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:36.658064 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:36.658237 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:36.658238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:36.658238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:36.658329 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:36.658530 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:36.658534 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:36.658642 (+ 108us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.658734 (+ 92us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.658845 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":92,"threads_started":1}
I20260926 05:41:36.659613 25436 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38685
I20260926 05:41:36.659732 25436 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:36.659915 25436 heartbeater.cc:511] Master 127.23.213.254:38685 requested a full tablet report, sending...
I20260926 05:41:36.660290 24977 ts_manager.cc:194] Registered new tserver with Master: f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 (127.23.213.195:38791)
I20260926 05:41:36.660616 24977 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:56781
I20260926 05:41:36.665441 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:41:36.672288 25016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.666856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40298 (local address 127.23.213.254:38685)
0926 05:41:36.666918 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.666920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.666990 (+ 70us) server_negotiation.cc:407] Connection header received
0926 05:41:36.667026 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.667026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.667034 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.667082 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.667579 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.668039 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.668766 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.668940 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.668991 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.670199 (+ 1208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.670200 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.670200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.670206 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.671370 (+ 1164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.671661 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.671662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.671662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.671719 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.671877 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.671877 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.671878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.672004 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.672074 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.672173 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.672228 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:41:36.674279 24977 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40298:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:41:36.675781 24977 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:36.681946 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.678030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46760 (local address 127.23.213.195:38791)
0926 05:41:36.678207 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.678212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.678355 (+ 143us) server_negotiation.cc:407] Connection header received
0926 05:41:36.678466 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.678469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.678486 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.678567 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:36.678745 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.679543 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.680885 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.681521 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.681607 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.681672 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.681831 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.681872 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":69,"threads_started":1}
I20260926 05:41:36.681939 25447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.677929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38791 (local address 127.0.0.1:46760)
0926 05:41:36.678253 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.678295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.678382 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.678590 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.678592 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.678594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:36.678664 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.678666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.679665 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.679666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.680792 (+ 1126us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.680794 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.681607 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.681607 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.681683 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.681697 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.681813 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.681869 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":78,"threads_started":1}
I20260926 05:41:36.682202 25451 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.677890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:46123 (local address 127.0.0.1:53292)
0926 05:41:36.678252 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.678295 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.678361 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.678667 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.678668 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.678670 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:36.678725 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.678726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.680089 (+ 1363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.680089 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.680996 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.680998 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.681964 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.681965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.682030 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.682035 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.682084 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.682121 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":331,"mutex_wait_us":49,"thread_start_us":69,"threads_started":1}
I20260926 05:41:36.682355 25448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.678115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:39577 (local address 127.0.0.1:58404)
0926 05:41:36.678253 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.678295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.678361 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.678712 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.678713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.678715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:36.678750 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.678751 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.680081 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.680082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.681201 (+ 1119us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.681203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.682087 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.682088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.682137 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.682140 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.682169 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.682215 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":31,"thread_start_us":81,"threads_started":1}
I20260926 05:41:36.682478 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.678238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58404 (local address 127.23.213.194:39577)
0926 05:41:36.678499 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.678503 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.678520 (+ 17us) server_negotiation.cc:407] Connection header received
0926 05:41:36.678576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.678578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.678595 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.678822 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:36.678904 (+ 82us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.679961 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.681283 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.681996 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.682264 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.682335 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.682362 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.682406 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":229,"thread_start_us":78,"threads_started":1}
I20260926 05:41:36.682535 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.678002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.23.213.193:46123)
0926 05:41:36.678147 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.678164 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.678449 (+ 285us) server_negotiation.cc:407] Connection header received
0926 05:41:36.678492 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.678494 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.678512 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.678577 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:36.678852 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.679983 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.681079 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.681807 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.681879 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.681929 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.682411 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.682454 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260926 05:41:36.683153 25370 tablet_service.cc:1467] Processing CreateTablet for tablet b820e3fe533d4178901fe21521a61d6c (DEFAULT_TABLE table=test-table [id=fd25da0b8aab44c1919ea7de3cb21c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:36.683152 25102 tablet_service.cc:1467] Processing CreateTablet for tablet b820e3fe533d4178901fe21521a61d6c (DEFAULT_TABLE table=test-table [id=fd25da0b8aab44c1919ea7de3cb21c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:36.683302 25236 tablet_service.cc:1467] Processing CreateTablet for tablet b820e3fe533d4178901fe21521a61d6c (DEFAULT_TABLE table=test-table [id=fd25da0b8aab44c1919ea7de3cb21c7f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:36.683490 25102 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b820e3fe533d4178901fe21521a61d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:36.683490 25370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b820e3fe533d4178901fe21521a61d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:36.683658 25236 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b820e3fe533d4178901fe21521a61d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:36.686233 25453 tablet_bootstrap.cc:492] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f: Bootstrap starting.
I20260926 05:41:36.686722 25455 tablet_bootstrap.cc:492] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6: Bootstrap starting.
I20260926 05:41:36.687062 25453 tablet_bootstrap.cc:654] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:36.687340 25455 tablet_bootstrap.cc:654] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:36.687657 25453 log.cc:824] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:36.687930 25455 log.cc:824] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:36.688246 25454 tablet_bootstrap.cc:492] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba: Bootstrap starting.
I20260926 05:41:36.688469 25453 tablet_bootstrap.cc:492] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f: No bootstrap required, opened a new log
I20260926 05:41:36.688546 25453 ts_tablet_manager.cc:1397] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:36.688663 25455 tablet_bootstrap.cc:492] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6: No bootstrap required, opened a new log
I20260926 05:41:36.688759 25455 ts_tablet_manager.cc:1397] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:36.688870 25454 tablet_bootstrap.cc:654] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:36.689429 25454 log.cc:824] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:36.690251 25454 tablet_bootstrap.cc:492] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba: No bootstrap required, opened a new log
I20260926 05:41:36.690330 25454 ts_tablet_manager.cc:1397] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:36.690622 25455 raft_consensus.cc:348] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.690793 25455 raft_consensus.cc:374] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:36.690836 25455 raft_consensus.cc:729] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2b26e8ef0ea4647bdf0d8b0f4e0ddb6, State: Initialized, Role: FOLLOWER
I20260926 05:41:36.690946 25455 consensus_queue.cc:260] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.691027 25453 raft_consensus.cc:348] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.691213 25453 raft_consensus.cc:374] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:36.691254 25455 ts_tablet_manager.cc:1428] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:41:36.691260 25453 raft_consensus.cc:729] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d98adac83fd941dba11b15f24a79827f, State: Initialized, Role: FOLLOWER
I20260926 05:41:36.691377 25436 heartbeater.cc:503] Master 127.23.213.254:38685 was elected leader, sending a full tablet report...
I20260926 05:41:36.691375 25453 consensus_queue.cc:260] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.691648 25453 ts_tablet_manager.cc:1428] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:41:36.691673 25302 heartbeater.cc:503] Master 127.23.213.254:38685 was elected leader, sending a full tablet report...
I20260926 05:41:36.692538 25454 raft_consensus.cc:348] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.692713 25454 raft_consensus.cc:374] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:36.692770 25454 raft_consensus.cc:729] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e1e877247a7a4d4aafa02de9e705e5ba, State: Initialized, Role: FOLLOWER
I20260926 05:41:36.692883 25454 consensus_queue.cc:260] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.693173 25454 ts_tablet_manager.cc:1428] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:41:36.693174 25168 heartbeater.cc:503] Master 127.23.213.254:38685 was elected leader, sending a full tablet report...
W20260926 05:41:36.745925 25303 tablet.cc:2369] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:41:36.844135 25169 tablet.cc:2369] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:41:36.881672 25461 raft_consensus.cc:482] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:36.881927 25461 raft_consensus.cc:504] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.882332 25461 leader_election.cc:288] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 (127.23.213.195:38791), d98adac83fd941dba11b15f24a79827f (127.23.213.194:39577)
I20260926 05:41:36.885363 25462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.882526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38791 (local address 127.23.213.193:53641)
0926 05:41:36.882690 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.882701 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.882793 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.882954 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.882955 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.882957 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:36.883037 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.883039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.883706 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.883707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.884544 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.884547 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.885123 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.885124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.885212 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.885216 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.885272 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.885303 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":53,"thread_start_us":86,"threads_started":1}
I20260926 05:41:36.885514 25450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.882628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:53641 (local address 127.23.213.195:38791)
0926 05:41:36.882683 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.882686 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.882782 (+ 96us) server_negotiation.cc:407] Connection header received
0926 05:41:36.882860 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.882861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.882869 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.882942 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:36.883113 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.883603 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.884676 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.885033 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.885089 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.885096 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.885401 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.885447 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:41:36.885588 25452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.882617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:42149 (local address 127.23.213.194:39577)
0926 05:41:36.882665 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.882667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.882784 (+ 117us) server_negotiation.cc:407] Connection header received
0926 05:41:36.882856 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.882857 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.882868 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.882953 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:36.883163 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.883608 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.884642 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.885098 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.885173 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.885181 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.885479 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.885519 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:41:36.885808 25463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.882528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:39577 (local address 127.23.213.193:42149)
0926 05:41:36.882694 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:36.882704 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:36.882793 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:36.882961 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:36.882961 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:36.882963 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:36.883062 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.883083 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.883720 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.883721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.884540 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:36.884542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.885200 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.885200 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:36.885384 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.885389 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:41:36.885656 (+ 267us) client_negotiation.cc:240] Negotiation successful
0926 05:41:36.885700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"mutex_wait_us":20,"thread_start_us":91,"threads_started":1}
I20260926 05:41:36.885854 25390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820e3fe533d4178901fe21521a61d6c" candidate_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" is_pre_election: true
I20260926 05:41:36.886027 25390 raft_consensus.cc:2454] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1e877247a7a4d4aafa02de9e705e5ba in term 0.
I20260926 05:41:36.886201 25256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820e3fe533d4178901fe21521a61d6c" candidate_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98adac83fd941dba11b15f24a79827f" is_pre_election: true
I20260926 05:41:36.886302 25056 leader_election.cc:302] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e1e877247a7a4d4aafa02de9e705e5ba, f2b26e8ef0ea4647bdf0d8b0f4e0ddb6; no voters:
I20260926 05:41:36.886356 25256 raft_consensus.cc:2454] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e1e877247a7a4d4aafa02de9e705e5ba in term 0.
I20260926 05:41:36.886538 25461 raft_consensus.cc:2781] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:36.886593 25461 raft_consensus.cc:482] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:36.886610 25461 raft_consensus.cc:3037] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:36.887269 25461 raft_consensus.cc:504] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.887408 25461 leader_election.cc:288] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [CANDIDATE]: Term 1 election: Requested vote from peers f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 (127.23.213.195:38791), d98adac83fd941dba11b15f24a79827f (127.23.213.194:39577)
I20260926 05:41:36.887552 25256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820e3fe533d4178901fe21521a61d6c" candidate_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d98adac83fd941dba11b15f24a79827f"
I20260926 05:41:36.887539 25390 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b820e3fe533d4178901fe21521a61d6c" candidate_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6"
I20260926 05:41:36.887620 25390 raft_consensus.cc:3037] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:36.887620 25256 raft_consensus.cc:3037] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:36.888474 25390 raft_consensus.cc:2454] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1e877247a7a4d4aafa02de9e705e5ba in term 1.
I20260926 05:41:36.888474 25256 raft_consensus.cc:2454] T b820e3fe533d4178901fe21521a61d6c P d98adac83fd941dba11b15f24a79827f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e1e877247a7a4d4aafa02de9e705e5ba in term 1.
I20260926 05:41:36.888649 25058 leader_election.cc:302] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d98adac83fd941dba11b15f24a79827f, e1e877247a7a4d4aafa02de9e705e5ba; no voters:
I20260926 05:41:36.888767 25461 raft_consensus.cc:2781] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:36.888902 25461 raft_consensus.cc:686] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [term 1 LEADER]: Becoming Leader. State: Replica: e1e877247a7a4d4aafa02de9e705e5ba, State: Running, Role: LEADER
I20260926 05:41:36.889008 25461 consensus_queue.cc:237] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } }
I20260926 05:41:36.889870 24975 catalog_manager.cc:5495] T b820e3fe533d4178901fe21521a61d6c P e1e877247a7a4d4aafa02de9e705e5ba reported cstate change: term changed from 0 to 1, leader changed from <none> to e1e877247a7a4d4aafa02de9e705e5ba (127.23.213.193). New cstate: current_term: 1 leader_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e1e877247a7a4d4aafa02de9e705e5ba" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46123 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2b26e8ef0ea4647bdf0d8b0f4e0ddb6" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d98adac83fd941dba11b15f24a79827f" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 39577 } health_report { overall_health: UNKNOWN } } }
W20260926 05:41:36.900142 25437 tablet.cc:2369] T b820e3fe533d4178901fe21521a61d6c P f2b26e8ef0ea4647bdf0d8b0f4e0ddb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401295, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:36.912228 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.904564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53294 (local address 127.23.213.193:46123)
0926 05:41:36.904634 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.904636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.904761 (+ 125us) server_negotiation.cc:407] Connection header received
0926 05:41:36.904835 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.904836 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.904852 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.904902 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.905482 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.906050 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.906769 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.906968 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.907056 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.909282 (+ 2226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.909294 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.909294 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.909301 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.910780 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.911227 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.911230 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.911230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.911331 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.911594 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.911595 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.911595 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.911794 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.911982 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.912018 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.912120 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:36.934505 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.927980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.23.213.193:46123)
0926 05:41:36.928045 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.928047 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.928055 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:41:36.928079 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.928079 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.928087 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.928134 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.928675 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.929137 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.929916 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.930099 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.930160 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.932422 (+ 2262us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.932424 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.932424 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.932430 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.933571 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.933930 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.933931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.933931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.933978 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.934124 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.934125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.934125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.934208 (+ 83us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.934272 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.934387 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.934444 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:41:36 dist-test-slave-1fdt krb5kdc[24932](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401296, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:36.957224 25449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:36.950517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53310 (local address 127.23.213.193:46123)
0926 05:41:36.950605 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:36.950607 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:36.950673 (+ 66us) server_negotiation.cc:407] Connection header received
0926 05:41:36.950713 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:36.950713 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:36.950723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:36.950772 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:36.951319 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.951781 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.952463 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:36.952637 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:36.952694 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:36.954865 (+ 2171us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:36.954866 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:36.954867 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:36.954873 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:36.956172 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.956441 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.956442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.956442 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.956490 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:36.956816 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:36.956817 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:36.956817 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:36.956904 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:36.956995 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:36.957078 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 05:41:36.957134 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
W20260926 05:41:36.957782 25082 tablet_service.cc:2908] Rejecting scan request for tablet b820e3fe533d4178901fe21521a61d6c: Uninitialized: clean time has not yet been initialized
I20260926 05:41:36.958673 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 25039
I20260926 05:41:36.963888 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 25173
I20260926 05:41:36.968377 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 25307
I20260926 05:41:36.972927 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 24945
2026-09-26T05:41:36Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2107 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:37 dist-test-slave-1fdt krb5kdc[25478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:37 dist-test-slave-1fdt krb5kdc[25478](info): set up 2 sockets
Sep 26 05:41:37 dist-test-slave-1fdt krb5kdc[25478](info): commencing operation
krb5kdc: starting...
W20260926 05:41:38.017472 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:38.137631 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40053
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:40053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:38.223013 25491 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:38.223220 25491 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:38.223242 25491 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:38.224463 25491 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:38.224511 25491 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:41:38.224525 25491 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:38.224567 25491 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:38.224596 25491 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:38.224612 25491 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:38.226090 25491 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:40053
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25491
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:38.226343 25491 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:38.226585 25491 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:38.231505 25496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:38.231503 25497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:38.231614 25499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:38.231745 25491 server_base.cc:1034] running on GCE node
I20260926 05:41:38.232014 25491 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:38.232311 25491 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:38.233489 25491 hybrid_clock.cc:648] HybridClock initialized: now 1790401298233471 us; error 54 us; skew 500 ppm
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:38.236012 25491 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:38.236495 25491 webserver.cc:460] Webserver started at http://127.23.213.254:42965/ using document root <none> and password file <none>
I20260926 05:41:38.236720 25491 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:38.236789 25491 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:38.236948 25491 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:38.238092 25491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "94d84d5071224e86b8e6a87aa9671516"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "353d6dba0a2a0a55314435b4761914bd"
server_key_iv: "ba2d8fb27c1e3769a3d9da07b75cade7"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.238461 25491 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "94d84d5071224e86b8e6a87aa9671516"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "353d6dba0a2a0a55314435b4761914bd"
server_key_iv: "ba2d8fb27c1e3769a3d9da07b75cade7"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.239809 25491 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:38.240485 25506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.240655 25491 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:38.240763 25491 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "94d84d5071224e86b8e6a87aa9671516"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "353d6dba0a2a0a55314435b4761914bd"
server_key_iv: "ba2d8fb27c1e3769a3d9da07b75cade7"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.240839 25491 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:38.251919 25491 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:38.254007 25491 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:38.254179 25491 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:38.259166 25491 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:40053
I20260926 05:41:38.259222 25568 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:40053 every 8 connection(s)
I20260926 05:41:38.259574 25491 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:38.260247 25569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:38.262491 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 25491
I20260926 05:41:38.262567 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:38.262601 25569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516: Bootstrap starting.
I20260926 05:41:38.262738 24407 external_mini_cluster.cc:1428] Setting key 1f1747902000207f1b6e1f9e5c333e97
I20260926 05:41:38.263381 25569 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:38.263728 25569 log.cc:824] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:38.264513 25569 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516: No bootstrap required, opened a new log
I20260926 05:41:38.266873 25569 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d84d5071224e86b8e6a87aa9671516" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40053 } }
I20260926 05:41:38.267083 25569 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:38.267156 25569 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94d84d5071224e86b8e6a87aa9671516, State: Initialized, Role: FOLLOWER
I20260926 05:41:38.267287 25569 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d84d5071224e86b8e6a87aa9671516" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40053 } }
I20260926 05:41:38.267359 25569 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:38.267380 25569 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:38.267446 25569 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:38.268124 25569 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d84d5071224e86b8e6a87aa9671516" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40053 } }
I20260926 05:41:38.268253 25569 leader_election.cc:302] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94d84d5071224e86b8e6a87aa9671516; no voters:
I20260926 05:41:38.268435 25569 leader_election.cc:288] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:38.268464 25574 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:38.268628 25574 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [term 1 LEADER]: Becoming Leader. State: Replica: 94d84d5071224e86b8e6a87aa9671516, State: Running, Role: LEADER
I20260926 05:41:38.268695 25569 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:38.268726 25574 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d84d5071224e86b8e6a87aa9671516" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40053 } }
I20260926 05:41:38.269074 25575 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94d84d5071224e86b8e6a87aa9671516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d84d5071224e86b8e6a87aa9671516" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40053 } } }
I20260926 05:41:38.269150 25575 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:38.269354 25576 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94d84d5071224e86b8e6a87aa9671516. Latest consensus state: current_term: 1 leader_uuid: "94d84d5071224e86b8e6a87aa9671516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94d84d5071224e86b8e6a87aa9671516" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40053 } } }
I20260926 05:41:38.269471 25576 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:38.269667 25579 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:38.270331 25579 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:38.272416 25579 catalog_manager.cc:1347] Generated new cluster ID: baf876b1cc204e308bd43c0cf2dc2894
I20260926 05:41:38.272469 25579 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:38.274537 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.263326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56230 (local address 127.23.213.254:40053)
0926 05:41:38.263513 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.263529 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.263558 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:41:38.264073 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.264080 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.264128 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.264245 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.264995 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.265960 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.267030 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.267650 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.267746 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.270833 (+ 3087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.270837 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.270838 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.270849 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.272859 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.273258 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.273261 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.273262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.273359 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.273580 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.273581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.273582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.273903 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.274086 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.274212 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.274356 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":78,"threads_started":1}
I20260926 05:41:38.283309 25579 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:38.283844 25579 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:38.288352 25579 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516: Generated new TSK 0
I20260926 05:41:38.288560 25579 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:38.315017 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40053
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:38.395527 25595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:38.395766 25595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:38.395805 25595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:38.396981 25595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:38.397047 25595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:38.397083 25595 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:41:38.397121 25595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:38.398587 25595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25595
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:38.398854 25595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:38.399080 25595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:38.404055 25600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:38.404031 25603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:38.404036 25601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:38.404220 25595 server_base.cc:1034] running on GCE node
I20260926 05:41:38.404518 25595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:38.404775 25595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:38.405956 25595 hybrid_clock.cc:648] HybridClock initialized: now 1790401298405949 us; error 35 us; skew 500 ppm
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:38.408330 25595 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:38.408787 25595 webserver.cc:460] Webserver started at http://127.23.213.193:44875/ using document root <none> and password file <none>
I20260926 05:41:38.409024 25595 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:38.409104 25595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:38.409257 25595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:38.410449 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "6efea551249b493ebeebd1c944328fad"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "366e4a9382716e7ec62996dac7bb61f4"
server_key_iv: "de1fc7e2c92b9e3423273f1b3d773c84"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.410830 25595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "6efea551249b493ebeebd1c944328fad"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "366e4a9382716e7ec62996dac7bb61f4"
server_key_iv: "de1fc7e2c92b9e3423273f1b3d773c84"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.412132 25595 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:38.412837 25610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.412999 25595 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:38.413105 25595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "6efea551249b493ebeebd1c944328fad"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "366e4a9382716e7ec62996dac7bb61f4"
server_key_iv: "de1fc7e2c92b9e3423273f1b3d773c84"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.413213 25595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:38.432195 25595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:38.434193 25595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:38.434369 25595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:38.434926 25595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:38.434993 25595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.435034 25595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:38.435077 25595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.442546 25595 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:35259
I20260926 05:41:38.442723 25725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:35259 every 8 connection(s)
I20260926 05:41:38.443022 25595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:38.446861 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.435485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:38321 (local address 127.23.213.254:40053)
0926 05:41:38.435563 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.435565 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.435880 (+ 315us) server_negotiation.cc:407] Connection header received
0926 05:41:38.436656 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.436657 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.436669 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.436781 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.437818 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.438802 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.440600 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.440840 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.440914 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.444170 (+ 3256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.444172 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.444173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.444182 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.445536 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.446021 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.446022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.446022 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.446078 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.446309 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.446309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.446309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.446422 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.446471 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.446720 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.446796 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:41:38.447091 25619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.435448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40053 (local address 127.23.213.193:38321)
0926 05:41:38.435683 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.435740 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.436522 (+ 782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.436882 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.436888 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.437336 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:38.437677 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.437686 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.438932 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.438934 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.440493 (+ 1559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.440498 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.440941 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.440942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.441118 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.441904 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:38.441917 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:38.444064 (+ 2147us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:38.445618 (+ 1554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.445625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.445627 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.445953 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.446138 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.446139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.446139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.446247 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.446474 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:38.446478 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:38.446606 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.446695 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.446825 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":107,"threads_started":1}
I20260926 05:41:38.447628 25726 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40053
I20260926 05:41:38.447723 25726 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:38.447914 25726 heartbeater.cc:511] Master 127.23.213.254:40053 requested a full tablet report, sending...
I20260926 05:41:38.448511 25522 ts_manager.cc:194] Registered new tserver with Master: 6efea551249b493ebeebd1c944328fad (127.23.213.193:35259)
I20260926 05:41:38.449401 25522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:38321
I20260926 05:41:38.450294 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 25595
I20260926 05:41:38.450398 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:38.450564 24407 external_mini_cluster.cc:1428] Setting key 1c4460b9a85b4454ec03bcf0ed914bde
I20260926 05:41:38.454757 25732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.448645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40053 (local address 127.23.213.193:44231)
0926 05:41:38.448791 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.448806 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.448866 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.448997 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.448997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.449150 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:38.449214 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.449216 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.450035 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.450036 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.450751 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.450753 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.451185 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.451186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.451275 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.451778 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:38.451783 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:38.452116 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:38.453797 (+ 1681us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.453799 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.453799 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.454003 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.454205 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.454206 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.454206 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.454251 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.454498 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:38.454499 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:38.454548 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.454594 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.454684 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260926 05:41:38.454870 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.448828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:44231 (local address 127.23.213.254:40053)
0926 05:41:38.448890 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.448892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.448900 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:41:38.448920 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.448921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.448930 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.448994 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.449279 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.449946 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.450837 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.451110 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.451186 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.452204 (+ 1018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.452205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.452206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.452213 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.453663 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.454075 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.454076 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.454076 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.454143 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.454304 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.454305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.454305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.454443 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.454510 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.454636 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.454733 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:38.476303 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40053
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:38.556212 25735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:38.556406 25735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:38.556429 25735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:38.557536 25735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:38.557569 25735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:38.557588 25735 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:41:38.557605 25735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:38.558960 25735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:38.559136 25735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:38.559309 25735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:38.564359 25743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:38.564366 25741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:38.564388 25735 server_base.cc:1034] running on GCE node
W20260926 05:41:38.564358 25740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:38.564831 25735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:38.565078 25735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:38.566247 25735 hybrid_clock.cc:648] HybridClock initialized: now 1790401298566237 us; error 27 us; skew 500 ppm
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:38.568621 25735 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:38.569105 25735 webserver.cc:460] Webserver started at http://127.23.213.194:43491/ using document root <none> and password file <none>
I20260926 05:41:38.569335 25735 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:38.569404 25735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:38.569577 25735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:38.570763 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "ea68056345cf46c994f5e10fdb9591c9"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "a7ef23240a6dfd6961888e5dd5e16c66"
server_key_iv: "0cfa79dbb6bdd9c47e528c5d57b8727d"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.571115 25735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "ea68056345cf46c994f5e10fdb9591c9"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "a7ef23240a6dfd6961888e5dd5e16c66"
server_key_iv: "0cfa79dbb6bdd9c47e528c5d57b8727d"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.572376 25735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:38.573079 25750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.573240 25735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:38.573338 25735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "ea68056345cf46c994f5e10fdb9591c9"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "a7ef23240a6dfd6961888e5dd5e16c66"
server_key_iv: "0cfa79dbb6bdd9c47e528c5d57b8727d"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.573417 25735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:38.584379 25735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:38.586339 25735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:38.586515 25735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:38.587097 25735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:38.587167 25735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.587206 25735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:38.587229 25735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:38.594712 25735 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:45033
I20260926 05:41:38.594801 25865 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:45033 every 8 connection(s)
I20260926 05:41:38.595137 25735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:41:38.597410 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.587519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:45543 (local address 127.23.213.254:40053)
0926 05:41:38.587591 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.587594 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.587891 (+ 297us) server_negotiation.cc:407] Connection header received
0926 05:41:38.588616 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.588618 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.588630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.588685 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.589637 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.590291 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.591981 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.592192 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.592271 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.594895 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.594896 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.594896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.594904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.596284 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.596678 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.596679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.596679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.596728 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.596931 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.596931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.596932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.597033 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.597081 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.597293 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.597349 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:41:38.597617 25759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.587516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40053 (local address 127.23.213.194:45543)
0926 05:41:38.587733 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.587792 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.588502 (+ 710us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.588766 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.588771 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.589201 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:38.589510 (+ 309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.589518 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.590401 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.590402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.591879 (+ 1477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.591882 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.592270 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.592272 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.592433 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.592932 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:38.592944 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:38.594793 (+ 1849us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:38.596353 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.596359 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.596360 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.596602 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.596783 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.596784 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.596784 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.596878 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.597085 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:38.597090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:38.597201 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.597284 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.597402 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":105,"threads_started":1}
I20260926 05:41:38.598174 25866 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40053
I20260926 05:41:38.598253 25866 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:38.598445 25866 heartbeater.cc:511] Master 127.23.213.254:40053 requested a full tablet report, sending...
I20260926 05:41:38.598853 25522 ts_manager.cc:194] Registered new tserver with Master: ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194:45033)
I20260926 05:41:38.599254 25522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:45543
I20260926 05:41:38.600944 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 25735
I20260926 05:41:38.601037 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:38.601209 24407 external_mini_cluster.cc:1428] Setting key 8dc5090e2047d7434ba2a477ffcb464c
I20260926 05:41:38.606455 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.599273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:45951 (local address 127.23.213.254:40053)
0926 05:41:38.599312 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.599313 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.599438 (+ 125us) server_negotiation.cc:407] Connection header received
0926 05:41:38.599499 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.599500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.599509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.599557 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.599874 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.600343 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.601323 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.601577 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.601633 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.603191 (+ 1558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.603193 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.603193 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.603200 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.605211 (+ 2011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.605656 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.605657 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.605657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.605707 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.605975 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.605976 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.605976 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.606111 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.606167 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.606339 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.606395 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:41:38.606566 25872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.599225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40053 (local address 127.23.213.194:45951)
0926 05:41:38.599369 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.599379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.599439 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.599564 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.599565 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.599736 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:38.599790 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.599791 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.600457 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.600458 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.601220 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.601223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.601654 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.601655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.601727 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.602659 (+ 932us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:38.602665 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:38.603094 (+ 429us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:38.605307 (+ 2213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.605308 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.605309 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.605570 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.605821 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.605822 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.605822 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.605905 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.606179 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:38.606180 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:38.606236 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.606321 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.606434 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:38.628978 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40053
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41407
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:38.709676 25875 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:38.709892 25875 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:38.709915 25875 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:38.711055 25875 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:38.711097 25875 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:38.711118 25875 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:41:38.711135 25875 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:38.712525 25875 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41407
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:40053
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25875
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:38.712713 25875 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:38.712890 25875 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:38.718214 25880 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:38.718181 25881 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:38.718392 25875 server_base.cc:1034] running on GCE node
W20260926 05:41:38.718168 25883 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:38.718719 25875 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:38.718992 25875 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:38.720139 25875 hybrid_clock.cc:648] HybridClock initialized: now 1790401298720125 us; error 30 us; skew 500 ppm
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:38.722703 25875 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:38.723199 25875 webserver.cc:460] Webserver started at http://127.23.213.195:41271/ using document root <none> and password file <none>
I20260926 05:41:38.723453 25875 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:38.723500 25875 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:38.723590 25875 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:38.724647 25875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "f521b60381af419691b270be272f1433"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "a26124ac14ff9e735ad44d5f58ddeb85"
server_key_iv: "cd71f584b7220165680ce980d5b2934c"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.724972 25875 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "f521b60381af419691b270be272f1433"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "a26124ac14ff9e735ad44d5f58ddeb85"
server_key_iv: "cd71f584b7220165680ce980d5b2934c"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.726392 25875 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:41:38.727118 25890 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.727301 25875 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:38.727411 25875 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "f521b60381af419691b270be272f1433"
format_stamp: "Formatted at 2026-09-26 05:41:38 on dist-test-slave-1fdt"
server_key: "a26124ac14ff9e735ad44d5f58ddeb85"
server_key_iv: "cd71f584b7220165680ce980d5b2934c"
server_key_version: "encryptionkey@0"
I20260926 05:41:38.727489 25875 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:38.735646 25875 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:38.737558 25875 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:38.737725 25875 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:38.738286 25875 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:38.738353 25875 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.738395 25875 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:38.738422 25875 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.745198 25875 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:33769
I20260926 05:41:38.745267 26005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:33769 every 8 connection(s)
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:38.745591 25875 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:38.748762 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.738821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:44973 (local address 127.23.213.254:40053)
0926 05:41:38.738896 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.738898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.739082 (+ 184us) server_negotiation.cc:407] Connection header received
0926 05:41:38.739567 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.739569 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.739583 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.739673 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.740497 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.741304 (+ 807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.742882 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.743123 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.743225 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.745849 (+ 2624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.745851 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.745851 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.745859 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.747282 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.747684 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.747685 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.747685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.747742 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.748027 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.748034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.748034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.748158 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.748229 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.748587 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.748668 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:41:38.748970 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.738791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40053 (local address 127.23.213.195:44973)
0926 05:41:38.738946 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.738986 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.739473 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.739727 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.739730 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.740115 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:38.740403 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.740409 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.741397 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.741398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.742780 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.742785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.743298 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.743299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.743449 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.743994 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:38.744004 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:38.745729 (+ 1725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:38.747369 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.747375 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.747377 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.747607 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.747830 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.747830 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.747831 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.747954 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.748230 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:38.748237 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:38.748427 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.748563 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.748727 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260926 05:41:38.750011 26006 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40053
I20260926 05:41:38.750093 26006 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:38.750275 26006 heartbeater.cc:511] Master 127.23.213.254:40053 requested a full tablet report, sending...
I20260926 05:41:38.750625 25523 ts_manager.cc:194] Registered new tserver with Master: f521b60381af419691b270be272f1433 (127.23.213.195:33769)
I20260926 05:41:38.751062 25523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:44973
I20260926 05:41:38.753868 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 25875
I20260926 05:41:38.753964 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:38.754164 24407 external_mini_cluster.cc:1428] Setting key 884b0e863ed5b45970fe677572f7c1af
I20260926 05:41:38.755105 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:41:38.756778 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.751132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:47363 (local address 127.23.213.254:40053)
0926 05:41:38.751170 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.751172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.751344 (+ 172us) server_negotiation.cc:407] Connection header received
0926 05:41:38.751404 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.751404 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.751414 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.751466 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.751819 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.752271 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.752913 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.753109 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.753156 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.754190 (+ 1034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.754191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.754191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.754196 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.755660 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.756001 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.756003 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.756003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.756069 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.756264 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.756265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.756265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.756374 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.756449 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.756613 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.756686 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:41:38.756793 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.751057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40053 (local address 127.23.213.195:47363)
0926 05:41:38.751274 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.751285 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.751338 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.751484 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.751484 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.751684 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:38.751749 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.751751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.752357 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.752358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.752849 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.752850 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.753172 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.753173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.753231 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.753601 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:38.753605 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:38.754108 (+ 503us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:38.755761 (+ 1653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.755762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.755763 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.755930 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.756136 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:38.756137 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:38.756137 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:38.756197 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:38.756440 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:38.756441 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:38.756497 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.756585 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.756688 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":80,"threads_started":1}
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:41:38 dist-test-slave-1fdt krb5kdc[25478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:38.777637 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.770995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56238 (local address 127.23.213.254:40053)
0926 05:41:38.771050 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.771052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.771078 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:41:38.771111 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.771111 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.771120 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.771166 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:38.771689 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.772177 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.772898 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.773074 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.773122 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.775329 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:38.775330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:38.775331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:38.775336 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:38.776717 (+ 1381us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.777001 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.777002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.777003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.777048 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:38.777261 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:38.777261 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:38.777262 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:38.777363 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:38.777430 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.777520 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.777573 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:41:38.779217 25523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:41:38.780773 25523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:38.787431 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.782968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:33769 (local address 127.0.0.1:37550)
0926 05:41:38.783177 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.783221 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.783287 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.783733 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.783736 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.783739 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:38.783829 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.783831 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.785271 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.785272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.786393 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.786395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.787111 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.787112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.787199 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.787214 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.787284 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.787343 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"mutex_wait_us":34,"thread_start_us":55,"threads_started":1}
I20260926 05:41:38.787528 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.783170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38348 (local address 127.23.213.193:35259)
0926 05:41:38.783544 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.783551 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.783600 (+ 49us) server_negotiation.cc:407] Connection header received
0926 05:41:38.783639 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.783642 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.783658 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.783906 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:38.783976 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.784872 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.786456 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.787162 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.787245 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.787300 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.787372 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.787408 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":342,"thread_start_us":72,"threads_started":1}
I20260926 05:41:38.787561 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.783184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45033 (local address 127.0.0.1:34344)
0926 05:41:38.783370 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.783376 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.783401 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.783735 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.783737 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.783741 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:38.783829 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.783832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.785311 (+ 1479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.785313 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.786288 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.786291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.787258 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.787259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.787326 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.787330 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.787364 (+ 34us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.787409 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":177,"mutex_wait_us":7,"thread_start_us":56,"threads_started":1}
I20260926 05:41:38.787788 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.783066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37550 (local address 127.23.213.195:33769)
0926 05:41:38.783518 (+ 452us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.783525 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.783534 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:41:38.783577 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.783579 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.783594 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.783910 (+ 316us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:38.783973 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.785160 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.786481 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.787016 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.787114 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.787176 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.787656 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.787700 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":117,"threads_started":1}
I20260926 05:41:38.787978 26023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.782968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:35259 (local address 127.0.0.1:38348)
0926 05:41:38.783368 (+ 400us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.783379 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.783433 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.783773 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.783774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.783776 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:38.783832 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.783834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.784972 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.784973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.786354 (+ 1381us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.786357 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.787244 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.787245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.787306 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.787309 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.787360 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.787398 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":386,"mutex_wait_us":3,"thread_start_us":57,"threads_started":1}
I20260926 05:41:38.788226 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.783366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34344 (local address 127.23.213.194:45033)
0926 05:41:38.783549 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.783553 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.783562 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:41:38.783597 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.783599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.783612 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.783910 (+ 298us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:38.783973 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.785203 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.786381 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.787162 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.787491 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.787747 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.787770 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.787802 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":82,"threads_started":1}
I20260926 05:41:38.788578 25940 tablet_service.cc:1467] Processing CreateTablet for tablet d34ef7876ef74c759738de2092b2398b (DEFAULT_TABLE table=test-table [id=7b696807ef9f4da49a78f24b78463974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:38.788795 25800 tablet_service.cc:1467] Processing CreateTablet for tablet d34ef7876ef74c759738de2092b2398b (DEFAULT_TABLE table=test-table [id=7b696807ef9f4da49a78f24b78463974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:38.788968 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34ef7876ef74c759738de2092b2398b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:38.789124 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34ef7876ef74c759738de2092b2398b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:38.791945 26026 tablet_bootstrap.cc:492] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Bootstrap starting.
I20260926 05:41:38.792200 25660 tablet_service.cc:1467] Processing CreateTablet for tablet d34ef7876ef74c759738de2092b2398b (DEFAULT_TABLE table=test-table [id=7b696807ef9f4da49a78f24b78463974]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:38.792507 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d34ef7876ef74c759738de2092b2398b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:38.792971 26026 tablet_bootstrap.cc:654] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:38.793283 26026 log.cc:824] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:38.793650 26027 tablet_bootstrap.cc:492] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Bootstrap starting.
I20260926 05:41:38.794063 26026 tablet_bootstrap.cc:492] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: No bootstrap required, opened a new log
I20260926 05:41:38.794135 26026 ts_tablet_manager.cc:1397] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:38.794332 26027 tablet_bootstrap.cc:654] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:38.794691 26027 log.cc:824] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:38.795502 26027 tablet_bootstrap.cc:492] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: No bootstrap required, opened a new log
I20260926 05:41:38.795580 26027 ts_tablet_manager.cc:1397] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:38.796242 26026 raft_consensus.cc:348] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.796414 26026 raft_consensus.cc:374] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:38.796489 26026 raft_consensus.cc:729] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f521b60381af419691b270be272f1433, State: Initialized, Role: FOLLOWER
I20260926 05:41:38.796634 26026 consensus_queue.cc:260] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.796891 26026 ts_tablet_manager.cc:1428] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:41:38.796932 26006 heartbeater.cc:503] Master 127.23.213.254:40053 was elected leader, sending a full tablet report...
I20260926 05:41:38.797446 26027 raft_consensus.cc:348] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.797595 26027 raft_consensus.cc:374] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:38.797641 26027 raft_consensus.cc:729] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea68056345cf46c994f5e10fdb9591c9, State: Initialized, Role: FOLLOWER
I20260926 05:41:38.797778 26027 consensus_queue.cc:260] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.798058 26027 ts_tablet_manager.cc:1428] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:41:38.798089 25866 heartbeater.cc:503] Master 127.23.213.254:40053 was elected leader, sending a full tablet report...
I20260926 05:41:38.798254 26030 tablet_bootstrap.cc:492] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Bootstrap starting.
I20260926 05:41:38.798894 26030 tablet_bootstrap.cc:654] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:38.799232 26030 log.cc:824] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:38.799913 26030 tablet_bootstrap.cc:492] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: No bootstrap required, opened a new log
I20260926 05:41:38.800022 26030 ts_tablet_manager.cc:1397] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:38.801334 26030 raft_consensus.cc:348] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.801456 26030 raft_consensus.cc:374] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:38.801486 26030 raft_consensus.cc:729] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efea551249b493ebeebd1c944328fad, State: Initialized, Role: FOLLOWER
I20260926 05:41:38.801595 26030 consensus_queue.cc:260] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.801879 26030 ts_tablet_manager.cc:1428] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:41:38.801918 25726 heartbeater.cc:503] Master 127.23.213.254:40053 was elected leader, sending a full tablet report...
W20260926 05:41:38.845815 25867 tablet.cc:2369] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:41:38.943818 25727 tablet.cc:2369] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:41:38.947360 26034 raft_consensus.cc:482] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:38.947513 26034 raft_consensus.cc:504] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.947890 26034 leader_election.cc:288] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f521b60381af419691b270be272f1433 (127.23.213.195:33769), ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194:45033)
I20260926 05:41:38.951062 26035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.948093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:33769 (local address 127.23.213.193:56009)
0926 05:41:38.948254 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.948264 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.948351 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.948507 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.948509 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.948511 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:38.948595 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.948596 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.949249 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.949250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.950242 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.950244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.950768 (+ 524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.950769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.950840 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.950849 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.950911 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.950951 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":39,"thread_start_us":70,"threads_started":1}
I20260926 05:41:38.951053 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.948255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:56009 (local address 127.23.213.195:33769)
0926 05:41:38.948317 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.948319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.948331 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:41:38.948419 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.948420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.948428 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.948501 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:38.948687 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.949163 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.950335 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.950695 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.950757 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.950766 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.950915 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.950951 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:41:38.951061 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.948179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:45527 (local address 127.23.213.194:45033)
0926 05:41:38.948225 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.948228 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.948306 (+ 78us) server_negotiation.cc:407] Connection header received
0926 05:41:38.948435 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.948436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.948445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.948502 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:38.948673 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.949163 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.950311 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.950691 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.950773 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.950781 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.950924 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.950959 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:41:38.951081 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.948092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45033 (local address 127.23.213.193:45527)
0926 05:41:38.948227 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.948239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.948306 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.948511 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.948512 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.948514 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:38.948596 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.948598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.949256 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.949257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.950208 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.950210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.950773 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.950774 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.950850 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.950853 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.950920 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.950960 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"mutex_wait_us":1,"thread_start_us":72,"threads_started":1}
I20260926 05:41:38.951411 25960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ef7876ef74c759738de2092b2398b" candidate_uuid: "6efea551249b493ebeebd1c944328fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f521b60381af419691b270be272f1433" is_pre_election: true
I20260926 05:41:38.951449 25820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ef7876ef74c759738de2092b2398b" candidate_uuid: "6efea551249b493ebeebd1c944328fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea68056345cf46c994f5e10fdb9591c9" is_pre_election: true
I20260926 05:41:38.951570 25820 raft_consensus.cc:2454] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6efea551249b493ebeebd1c944328fad in term 0.
I20260926 05:41:38.951570 25960 raft_consensus.cc:2454] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6efea551249b493ebeebd1c944328fad in term 0.
I20260926 05:41:38.951785 25614 leader_election.cc:302] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efea551249b493ebeebd1c944328fad, f521b60381af419691b270be272f1433; no voters:
I20260926 05:41:38.951934 26034 raft_consensus.cc:2781] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:38.951988 26034 raft_consensus.cc:482] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:38.952013 26034 raft_consensus.cc:3037] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:38.952732 26034 raft_consensus.cc:504] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.952886 26034 leader_election.cc:288] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [CANDIDATE]: Term 1 election: Requested vote from peers f521b60381af419691b270be272f1433 (127.23.213.195:33769), ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194:45033)
I20260926 05:41:38.953065 25960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ef7876ef74c759738de2092b2398b" candidate_uuid: "6efea551249b493ebeebd1c944328fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f521b60381af419691b270be272f1433"
I20260926 05:41:38.953066 25820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d34ef7876ef74c759738de2092b2398b" candidate_uuid: "6efea551249b493ebeebd1c944328fad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea68056345cf46c994f5e10fdb9591c9"
I20260926 05:41:38.953143 25820 raft_consensus.cc:3037] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:38.953143 25960 raft_consensus.cc:3037] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:38.954047 25820 raft_consensus.cc:2454] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efea551249b493ebeebd1c944328fad in term 1.
I20260926 05:41:38.954047 25960 raft_consensus.cc:2454] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6efea551249b493ebeebd1c944328fad in term 1.
I20260926 05:41:38.954275 25611 leader_election.cc:302] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efea551249b493ebeebd1c944328fad, ea68056345cf46c994f5e10fdb9591c9; no voters:
I20260926 05:41:38.954380 26034 raft_consensus.cc:2781] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:38.954550 26034 raft_consensus.cc:686] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 1 LEADER]: Becoming Leader. State: Replica: 6efea551249b493ebeebd1c944328fad, State: Running, Role: LEADER
I20260926 05:41:38.954658 26034 consensus_queue.cc:237] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } }
I20260926 05:41:38.955435 25523 catalog_manager.cc:5495] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad reported cstate change: term changed from 0 to 1, leader changed from <none> to 6efea551249b493ebeebd1c944328fad (127.23.213.193). New cstate: current_term: 1 leader_uuid: "6efea551249b493ebeebd1c944328fad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:38.968094 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.965846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38362 (local address 127.23.213.193:35259)
0926 05:41:38.965907 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.965909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.965954 (+ 45us) server_negotiation.cc:407] Connection header received
0926 05:41:38.966056 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.966057 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.966072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.966132 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:38.966263 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.966720 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.967426 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.967608 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.967658 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.967831 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:38.967921 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:38.967976 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.968020 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.968039 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:41:38.969849 25960 raft_consensus.cc:1261] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Refusing update from remote peer 6efea551249b493ebeebd1c944328fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:38.969849 25820 raft_consensus.cc:1261] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Refusing update from remote peer 6efea551249b493ebeebd1c944328fad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:38.970218 26037 consensus_queue.cc:1035] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:38.970382 26034 consensus_queue.cc:1035] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [LEADER]: Connected to new peer: Peer: permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:38.973277 26041 mvcc.cc:205] Tried to move back new op lower bound from 7333483720577630208 to 7333483720519032832. Current Snapshot: MvccSnapshot[applied={T|T < 7333483720577630208 or (T in {7333483720577630208})}]
I20260926 05:41:38.973646 26040 mvcc.cc:205] Tried to move back new op lower bound from 7333483720577630208 to 7333483720519032832. Current Snapshot: MvccSnapshot[applied={T|T < 7333483720577630208 or (T in {7333483720577630208})}]
I20260926 05:41:38.973706 26042 mvcc.cc:205] Tried to move back new op lower bound from 7333483720577630208 to 7333483720519032832. Current Snapshot: MvccSnapshot[applied={T|T < 7333483720577630208}]
I20260926 05:41:38.976116 25523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:38.976248 25523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:38.976958 25523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516: Sending DeleteTablet for 3 replicas of tablet d34ef7876ef74c759738de2092b2398b
I20260926 05:41:38.977291 25940 tablet_service.cc:1514] Processing DeleteTablet for tablet d34ef7876ef74c759738de2092b2398b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:38 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37550
I20260926 05:41:38.977329 25800 tablet_service.cc:1514] Processing DeleteTablet for tablet d34ef7876ef74c759738de2092b2398b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:38 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:34344
I20260926 05:41:38.977326 25660 tablet_service.cc:1514] Processing DeleteTablet for tablet d34ef7876ef74c759738de2092b2398b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:38 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:38348
I20260926 05:41:38.977739 26051 tablet_replica.cc:330] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: stopping tablet replica
I20260926 05:41:38.977790 26052 tablet_replica.cc:330] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: stopping tablet replica
I20260926 05:41:38.977880 26052 raft_consensus.cc:2229] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:38.977880 26051 raft_consensus.cc:2229] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:41:38.977854 25523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56238:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:41:38.977982 26052 raft_consensus.cc:2258] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:38.977980 26051 raft_consensus.cc:2258] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 05:41:38.978012 25523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:38.978462 26052 ts_tablet_manager.cc:1905] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:38.979051 26051 ts_tablet_manager.cc:1905] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:38.979725 25660 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae18d064104d7fb91f14a25f7c7753 (DEFAULT_TABLE table=test-table [id=38e6defd595e40488d5b122e27ac298c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:38.979837 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae18d064104d7fb91f14a25f7c7753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:38.978133 26050 tablet_replica.cc:330] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: stopping tablet replica
I20260926 05:41:38.980497 26050 raft_consensus.cc:2229] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:38.980611 26050 raft_consensus.cc:2258] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:38.981163 26051 ts_tablet_manager.cc:1918] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:41:38.981166 26030 tablet_bootstrap.cc:492] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: Bootstrap starting.
I20260926 05:41:38.981208 25800 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae18d064104d7fb91f14a25f7c7753 (DEFAULT_TABLE table=test-table [id=38e6defd595e40488d5b122e27ac298c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:38.981246 26051 log.cc:1196] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/d34ef7876ef74c759738de2092b2398b
I20260926 05:41:38.979496 25940 tablet_service.cc:1467] Processing CreateTablet for tablet 38ae18d064104d7fb91f14a25f7c7753 (DEFAULT_TABLE table=test-table [id=38e6defd595e40488d5b122e27ac298c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:38.981523 26051 ts_tablet_manager.cc:1939] T d34ef7876ef74c759738de2092b2398b P 6efea551249b493ebeebd1c944328fad: Deleting consensus metadata
I20260926 05:41:38.981551 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae18d064104d7fb91f14a25f7c7753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:38.981611 26030 tablet_bootstrap.cc:654] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:38.982271 26030 tablet_bootstrap.cc:492] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: No bootstrap required, opened a new log
I20260926 05:41:38.982314 26030 ts_tablet_manager.cc:1397] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:38.982461 26030 raft_consensus.cc:348] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:38.982525 26030 raft_consensus.cc:374] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:38.982540 26030 raft_consensus.cc:729] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efea551249b493ebeebd1c944328fad, State: Initialized, Role: FOLLOWER
I20260926 05:41:38.982583 26030 consensus_queue.cc:260] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:38.982636 26030 ts_tablet_manager.cc:1428] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.982910 26026 tablet_bootstrap.cc:492] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Bootstrap starting.
I20260926 05:41:38.983286 26026 tablet_bootstrap.cc:654] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:38.983646 25507 catalog_manager.cc:4841] TS 6efea551249b493ebeebd1c944328fad (127.23.213.193:35259): tablet d34ef7876ef74c759738de2092b2398b (table test-table [id=7b696807ef9f4da49a78f24b78463974]) successfully deleted
I20260926 05:41:38.984140 26026 tablet_bootstrap.cc:492] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: No bootstrap required, opened a new log
I20260926 05:41:38.984195 26026 ts_tablet_manager.cc:1397] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:38.984324 26026 raft_consensus.cc:348] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:38.984402 26026 raft_consensus.cc:374] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:38.984424 26026 raft_consensus.cc:729] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f521b60381af419691b270be272f1433, State: Initialized, Role: FOLLOWER
I20260926 05:41:38.984463 26026 consensus_queue.cc:260] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:38.984547 26026 ts_tablet_manager.cc:1428] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:38.985034 26050 ts_tablet_manager.cc:1905] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:38.985896 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38ae18d064104d7fb91f14a25f7c7753. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:38.986313 26050 ts_tablet_manager.cc:1918] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:41:38.986402 26050 log.cc:1196] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/d34ef7876ef74c759738de2092b2398b
I20260926 05:41:38.986631 26050 ts_tablet_manager.cc:1939] T d34ef7876ef74c759738de2092b2398b P f521b60381af419691b270be272f1433: Deleting consensus metadata
I20260926 05:41:38.987010 25510 catalog_manager.cc:4841] TS f521b60381af419691b270be272f1433 (127.23.213.195:33769): tablet d34ef7876ef74c759738de2092b2398b (table test-table [id=7b696807ef9f4da49a78f24b78463974]) successfully deleted
I20260926 05:41:38.987159 26027 tablet_bootstrap.cc:492] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: Bootstrap starting.
I20260926 05:41:38.987253 26052 ts_tablet_manager.cc:1918] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:41:38.987323 26052 log.cc:1196] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/d34ef7876ef74c759738de2092b2398b
I20260926 05:41:38.987565 26027 tablet_bootstrap.cc:654] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:38.987591 26052 ts_tablet_manager.cc:1939] T d34ef7876ef74c759738de2092b2398b P ea68056345cf46c994f5e10fdb9591c9: Deleting consensus metadata
I20260926 05:41:38.988025 25507 catalog_manager.cc:4841] TS ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194:45033): tablet d34ef7876ef74c759738de2092b2398b (table test-table [id=7b696807ef9f4da49a78f24b78463974]) successfully deleted
I20260926 05:41:38.988166 26027 tablet_bootstrap.cc:492] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: No bootstrap required, opened a new log
I20260926 05:41:38.988224 26027 ts_tablet_manager.cc:1397] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:38.988375 26027 raft_consensus.cc:348] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:38.988452 26027 raft_consensus.cc:374] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:38.988485 26027 raft_consensus.cc:729] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea68056345cf46c994f5e10fdb9591c9, State: Initialized, Role: FOLLOWER
I20260926 05:41:38.988533 26027 consensus_queue.cc:260] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:38.988617 26027 ts_tablet_manager.cc:1428] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:38.994897 26031 raft_consensus.cc:482] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:38.994978 26031 raft_consensus.cc:504] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:38.995280 26031 leader_election.cc:288] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efea551249b493ebeebd1c944328fad (127.23.213.193:35259), ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194:45033)
W20260926 05:41:38.996237 26007 tablet.cc:2369] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:41:38.998625 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.995712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:49155 (local address 127.23.213.194:45033)
0926 05:41:38.995753 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.995755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.995764 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:41:38.995827 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.995828 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.995836 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.995885 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:38.996096 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.996669 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.997809 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.998267 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.998339 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.998348 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.998505 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.998541 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:41:38.998631 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.995555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45033 (local address 127.23.213.195:49155)
0926 05:41:38.995715 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.995725 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.995777 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.995930 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.995931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.995933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:38.996009 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.996016 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.996757 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.996758 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.997683 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.997686 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.998348 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.998349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.998425 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.998429 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.998489 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.998531 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"mutex_wait_us":11,"thread_start_us":48,"threads_started":1}
I20260926 05:41:38.998939 25820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae18d064104d7fb91f14a25f7c7753" candidate_uuid: "f521b60381af419691b270be272f1433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea68056345cf46c994f5e10fdb9591c9" is_pre_election: true
I20260926 05:41:38.999022 25820 raft_consensus.cc:2454] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f521b60381af419691b270be272f1433 in term 0.
I20260926 05:41:38.999063 25899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.995452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:35259 (local address 127.23.213.195:52225)
0926 05:41:38.995505 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:38.995519 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:38.995572 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:38.995793 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:38.995794 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:38.995796 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:38.995855 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.995857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.996774 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.996775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.998013 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:38.998016 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.998779 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.998781 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:38.998878 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.998882 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:38.998945 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:41:38.998992 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":40}
I20260926 05:41:38.999194 25891 leader_election.cc:302] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea68056345cf46c994f5e10fdb9591c9, f521b60381af419691b270be272f1433; no voters:
I20260926 05:41:38.999258 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:38.995531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:52225 (local address 127.23.213.193:35259)
0926 05:41:38.995577 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:38.995579 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:38.995587 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:41:38.995687 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:38.995688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:38.995698 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:38.995747 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:38.995924 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.996679 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.998101 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:38.998641 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:38.998720 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:38.998728 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:38.999146 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 05:41:38.999197 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:41:38.999359 26031 raft_consensus.cc:2781] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:38.999399 26031 raft_consensus.cc:482] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:38.999419 26031 raft_consensus.cc:3037] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:39.000276 26031 raft_consensus.cc:504] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.000403 26031 leader_election.cc:288] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [CANDIDATE]: Term 1 election: Requested vote from peers 6efea551249b493ebeebd1c944328fad (127.23.213.193:35259), ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194:45033)
I20260926 05:41:39.000644 25680 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae18d064104d7fb91f14a25f7c7753" candidate_uuid: "f521b60381af419691b270be272f1433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efea551249b493ebeebd1c944328fad" is_pre_election: true
I20260926 05:41:39.000777 25679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae18d064104d7fb91f14a25f7c7753" candidate_uuid: "f521b60381af419691b270be272f1433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efea551249b493ebeebd1c944328fad"
I20260926 05:41:39.000799 25680 raft_consensus.cc:2454] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f521b60381af419691b270be272f1433 in term 0.
I20260926 05:41:39.000886 25820 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38ae18d064104d7fb91f14a25f7c7753" candidate_uuid: "f521b60381af419691b270be272f1433" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea68056345cf46c994f5e10fdb9591c9"
I20260926 05:41:39.000950 25820 raft_consensus.cc:3037] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:39.001641 25820 raft_consensus.cc:2454] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f521b60381af419691b270be272f1433 in term 1.
I20260926 05:41:39.001853 25891 leader_election.cc:302] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ea68056345cf46c994f5e10fdb9591c9, f521b60381af419691b270be272f1433; no voters: 6efea551249b493ebeebd1c944328fad
I20260926 05:41:39.001993 26031 raft_consensus.cc:2781] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:39.002137 26031 raft_consensus.cc:686] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 1 LEADER]: Becoming Leader. State: Replica: f521b60381af419691b270be272f1433, State: Running, Role: LEADER
I20260926 05:41:39.002206 26031 consensus_queue.cc:237] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.002884 25523 catalog_manager.cc:5495] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 reported cstate change: term changed from 0 to 1, leader changed from <none> to f521b60381af419691b270be272f1433 (127.23.213.195). New cstate: current_term: 1 leader_uuid: "f521b60381af419691b270be272f1433" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } health_report { overall_health: HEALTHY } } }
Sep 26 05:41:39 dist-test-slave-1fdt krb5kdc[25478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:39.042091 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.034501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40053 (local address 127.0.0.1:56250)
0926 05:41:39.034704 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:39.034718 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:39.034797 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:39.034947 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:39.034948 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:39.035347 (+ 399us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:39.035406 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.035409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.036053 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.036054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.036896 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.036898 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.037221 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.037222 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.037280 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.037865 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:39.037870 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:39.039384 (+ 1514us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:39.040933 (+ 1549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:39.040940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:39.040943 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:39.041135 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:39.041357 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:39.041358 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:39.041359 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:39.041416 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:39.041657 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:39.041664 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:39.041716 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 05:41:39.041825 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:41:39.041956 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":2,"thread_start_us":111,"threads_started":1}
I20260926 05:41:39.042136 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.034548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56250 (local address 127.23.213.254:40053)
0926 05:41:39.034617 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.034620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.034791 (+ 171us) server_negotiation.cc:407] Connection header received
0926 05:41:39.034860 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.034860 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.034869 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.034920 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:39.035484 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.035976 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.036975 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.037148 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.037196 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.039480 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:39.039481 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:39.039482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:39.039488 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:39.040847 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:39.041234 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:39.041235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:39.041236 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:39.041290 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:39.041484 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:39.041485 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:39.041485 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:39.041591 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:39.041665 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.041850 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.042026 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":97,"server-negotiator.queue_time_us":42}
I20260926 05:41:39.044552 25523 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:56250:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:41:39.044770 25523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:39.046271 25800 tablet_service.cc:1467] Processing CreateTablet for tablet 9acce70bf7584ed59dd59913a51ac303 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7310d47cceb4a27bc7203e8e2f242b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:41:39.046300 25660 tablet_service.cc:1467] Processing CreateTablet for tablet 9acce70bf7584ed59dd59913a51ac303 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7310d47cceb4a27bc7203e8e2f242b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:41:39.046320 25940 tablet_service.cc:1467] Processing CreateTablet for tablet 9acce70bf7584ed59dd59913a51ac303 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=f7310d47cceb4a27bc7203e8e2f242b0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:41:39.046432 25800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acce70bf7584ed59dd59913a51ac303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:39.046437 25940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acce70bf7584ed59dd59913a51ac303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:39.046442 25660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9acce70bf7584ed59dd59913a51ac303. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:39.047904 26030 tablet_bootstrap.cc:492] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: Bootstrap starting.
I20260926 05:41:39.047974 26027 tablet_bootstrap.cc:492] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: Bootstrap starting.
I20260926 05:41:39.048254 26030 tablet_bootstrap.cc:654] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:39.048346 26027 tablet_bootstrap.cc:654] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:39.048609 26026 tablet_bootstrap.cc:492] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: Bootstrap starting.
I20260926 05:41:39.048955 26027 tablet_bootstrap.cc:492] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: No bootstrap required, opened a new log
I20260926 05:41:39.048988 26027 ts_tablet_manager.cc:1397] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:39.049013 26026 tablet_bootstrap.cc:654] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:39.049116 26027 raft_consensus.cc:348] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.049163 26027 raft_consensus.cc:374] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:39.049178 26027 raft_consensus.cc:729] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea68056345cf46c994f5e10fdb9591c9, State: Initialized, Role: FOLLOWER
I20260926 05:41:39.049201 26027 consensus_queue.cc:260] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.049296 26027 ts_tablet_manager.cc:1428] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:39.049362 26032 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.049436 26032 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:39.050109 26026 tablet_bootstrap.cc:492] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: No bootstrap required, opened a new log
I20260926 05:41:39.050164 26026 ts_tablet_manager.cc:1397] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:39.050302 26026 raft_consensus.cc:348] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.050374 26026 raft_consensus.cc:374] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:39.050395 26026 raft_consensus.cc:729] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f521b60381af419691b270be272f1433, State: Initialized, Role: FOLLOWER
I20260926 05:41:39.050434 26026 consensus_queue.cc:260] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.050527 26026 ts_tablet_manager.cc:1428] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:39.050590 26031 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.050679 26031 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:39.051110 26030 tablet_bootstrap.cc:492] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: No bootstrap required, opened a new log
I20260926 05:41:39.051164 26030 ts_tablet_manager.cc:1397] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260926 05:41:39.051335 26030 raft_consensus.cc:348] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.051415 26030 raft_consensus.cc:374] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:39.051440 26030 raft_consensus.cc:729] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6efea551249b493ebeebd1c944328fad, State: Initialized, Role: FOLLOWER
I20260926 05:41:39.051491 26030 consensus_queue.cc:260] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.051581 26030 ts_tablet_manager.cc:1428] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:39.051616 26047 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.051731 26047 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:39.154049 26032 raft_consensus.cc:482] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:39.154189 26032 raft_consensus.cc:504] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.154609 26032 leader_election.cc:288] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6efea551249b493ebeebd1c944328fad (127.23.213.193:35259), f521b60381af419691b270be272f1433 (127.23.213.195:33769)
I20260926 05:41:39.157699 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.154902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:52563 (local address 127.23.213.195:33769)
0926 05:41:39.154974 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.154976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.155112 (+ 136us) server_negotiation.cc:407] Connection header received
0926 05:41:39.155201 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.155201 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.155211 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.155268 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:39.155448 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.155900 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.156960 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.157327 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.157382 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.157389 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.157574 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.157618 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:41:39.157719 26069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.154787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:33769 (local address 127.23.213.194:52563)
0926 05:41:39.155007 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:39.155037 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:39.155111 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:39.155283 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:39.155284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:39.155286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:39.155373 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.155375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.156019 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.156020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.156838 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.156840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.157406 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.157407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.157486 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.157491 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:41:39.157560 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:41:39.157593 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260926 05:41:39.157835 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.154868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:35657 (local address 127.23.213.193:35259)
0926 05:41:39.154939 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.154941 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.155043 (+ 102us) server_negotiation.cc:407] Connection header received
0926 05:41:39.155104 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.155105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.155116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.155172 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:39.155363 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.155915 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.156940 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.157404 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.157490 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.157499 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.157684 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.157731 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:41:39.157974 26068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.154787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:35259 (local address 127.23.213.194:35657)
0926 05:41:39.154948 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:39.154961 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:39.155039 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:39.155199 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:39.155200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:39.155202 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:39.155285 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.155287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.156090 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.156091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.156838 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.156840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.157497 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.157498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.157597 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.157601 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:39.157666 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:41:39.157705 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260926 05:41:39.158169 25960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acce70bf7584ed59dd59913a51ac303" candidate_uuid: "ea68056345cf46c994f5e10fdb9591c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f521b60381af419691b270be272f1433" is_pre_election: true
I20260926 05:41:39.158221 25679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acce70bf7584ed59dd59913a51ac303" candidate_uuid: "ea68056345cf46c994f5e10fdb9591c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efea551249b493ebeebd1c944328fad" is_pre_election: true
I20260926 05:41:39.158273 25960 raft_consensus.cc:2454] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea68056345cf46c994f5e10fdb9591c9 in term 0.
I20260926 05:41:39.158313 25679 raft_consensus.cc:2454] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ea68056345cf46c994f5e10fdb9591c9 in term 0.
I20260926 05:41:39.158447 25754 leader_election.cc:302] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ea68056345cf46c994f5e10fdb9591c9, f521b60381af419691b270be272f1433; no voters:
I20260926 05:41:39.158584 26032 raft_consensus.cc:2781] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:39.158629 26032 raft_consensus.cc:482] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:39.158653 26032 raft_consensus.cc:3037] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:39.159358 26032 raft_consensus.cc:504] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.159511 26032 leader_election.cc:288] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [CANDIDATE]: Term 1 election: Requested vote from peers 6efea551249b493ebeebd1c944328fad (127.23.213.193:35259), f521b60381af419691b270be272f1433 (127.23.213.195:33769)
I20260926 05:41:39.159668 25679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acce70bf7584ed59dd59913a51ac303" candidate_uuid: "ea68056345cf46c994f5e10fdb9591c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6efea551249b493ebeebd1c944328fad"
I20260926 05:41:39.159776 25679 raft_consensus.cc:3037] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:39.160800 25679 raft_consensus.cc:2454] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea68056345cf46c994f5e10fdb9591c9 in term 1.
I20260926 05:41:39.160800 25960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9acce70bf7584ed59dd59913a51ac303" candidate_uuid: "ea68056345cf46c994f5e10fdb9591c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f521b60381af419691b270be272f1433"
I20260926 05:41:39.160873 25960 raft_consensus.cc:3037] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:39.160992 25751 leader_election.cc:302] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6efea551249b493ebeebd1c944328fad, ea68056345cf46c994f5e10fdb9591c9; no voters:
I20260926 05:41:39.161082 26032 raft_consensus.cc:2781] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:39.161203 26032 raft_consensus.cc:686] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [term 1 LEADER]: Becoming Leader. State: Replica: ea68056345cf46c994f5e10fdb9591c9, State: Running, Role: LEADER
I20260926 05:41:39.161286 26032 consensus_queue.cc:237] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } }
I20260926 05:41:39.161586 26070 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: TxnStatusTablet state changed. Reason: New leader ea68056345cf46c994f5e10fdb9591c9. Latest consensus state: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.161660 26070 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:41:39.161861 26072 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:41:39.161952 26072 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:41:39.162204 25960 raft_consensus.cc:2454] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea68056345cf46c994f5e10fdb9591c9 in term 1.
I20260926 05:41:39.162477 25523 catalog_manager.cc:5495] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 reported cstate change: term changed from 0 to 1, leader changed from <none> to ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:39.169838 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.166944 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45033 (local address 127.0.0.1:34354)
0926 05:41:39.167082 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:39.167095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:39.167157 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:39.167305 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:39.167306 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:39.167308 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:39.167359 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.167362 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.167940 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.167941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.168891 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.168895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.169241 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.169243 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.169317 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.169336 (+ 19us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:39.169593 (+ 257us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:39.169621 (+ 28us) client_negotiation.cc:769] Sending connection context
0926 05:41:39.169686 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 05:41:39.169757 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 05:41:39.169878 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.167005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34354 (local address 127.23.213.194:45033)
0926 05:41:39.167056 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.167058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.167135 (+ 77us) server_negotiation.cc:407] Connection header received
0926 05:41:39.167232 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.167233 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.167247 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.167300 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:39.167420 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.167860 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.168975 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.169161 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.169215 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.169430 (+ 215us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:39.169519 (+ 89us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:39.169573 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.169690 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.169733 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:41:39.170946 25960 raft_consensus.cc:1261] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Refusing update from remote peer ea68056345cf46c994f5e10fdb9591c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:39.170985 25679 raft_consensus.cc:1261] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Refusing update from remote peer ea68056345cf46c994f5e10fdb9591c9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:39.171212 26070 consensus_queue.cc:1035] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:39.171348 26032 consensus_queue.cc:1035] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:39.171972 26031 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: TxnStatusTablet state changed. Reason: New leader ea68056345cf46c994f5e10fdb9591c9. Latest consensus state: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.172056 26031 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:39.172017 26047 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: TxnStatusTablet state changed. Reason: New leader ea68056345cf46c994f5e10fdb9591c9. Latest consensus state: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.172106 26047 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:39.172883 26032 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.172973 26032 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:41:39.173333 26031 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.173436 26031 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P f521b60381af419691b270be272f1433: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:39.174366 26032 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.174495 26032 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P ea68056345cf46c994f5e10fdb9591c9: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:41:39.174669 26047 tablet_replica.cc:439] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea68056345cf46c994f5e10fdb9591c9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 } } peers { permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 } } peers { permanent_uuid: "f521b60381af419691b270be272f1433" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33769 } } }
I20260926 05:41:39.174753 26047 tablet_replica.cc:442] T 9acce70bf7584ed59dd59913a51ac303 P 6efea551249b493ebeebd1c944328fad: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:39.177280 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.174723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37564 (local address 127.23.213.195:33769)
0926 05:41:39.174762 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.174764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.174803 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:41:39.174881 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.174882 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.174903 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.174964 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:39.175130 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.175728 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.176468 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.176679 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.176763 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.176933 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:39.177041 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:39.177116 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.177164 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.177184 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:41:39.182169 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.179934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:55487 (local address 127.23.213.194:45033)
0926 05:41:39.179990 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.179992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.179998 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:41:39.180024 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.180025 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.180033 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.180095 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:39.180238 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.180695 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.181424 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.181615 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.181711 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.181931 (+ 220us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:39.181968 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:39.182029 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.182082 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.182102 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:41:39.182200 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.179733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45033 (local address 127.23.213.195:55487)
0926 05:41:39.179812 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:39.179822 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:39.179869 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:39.180099 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:39.180100 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:39.180101 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:39.180163 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.180165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.180781 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.180782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.181333 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.181334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.181714 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.181715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.181818 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.181834 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:39.182031 (+ 197us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:39.182033 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:41:39.182080 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 05:41:39.182121 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 05:41:39.184479 25820 raft_consensus.cc:1261] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Refusing update from remote peer f521b60381af419691b270be272f1433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:39.184490 25679 raft_consensus.cc:3037] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:39.185262 25679 raft_consensus.cc:1261] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Refusing update from remote peer f521b60381af419691b270be272f1433: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:39.185470 26031 consensus_queue.cc:1035] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6efea551249b493ebeebd1c944328fad" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:39.185617 26057 consensus_queue.cc:1035] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea68056345cf46c994f5e10fdb9591c9" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:39.194854 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.191921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:52785 (local address 127.23.213.195:33769)
0926 05:41:39.191991 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.191994 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.192150 (+ 156us) server_negotiation.cc:407] Connection header received
0926 05:41:39.192189 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.192190 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.192201 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.192275 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:39.192372 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.193096 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.194161 (+ 1065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.194366 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.194443 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.194625 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:39.194659 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:39.194707 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.194762 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.194781 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:41:39.194896 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.191867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:33769 (local address 127.23.213.194:52785)
0926 05:41:39.192074 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:39.192086 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:39.192142 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:39.192275 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:39.192275 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:39.192277 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:39.192330 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.192332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.193173 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.193174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.194088 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:39.194091 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.194441 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.194442 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:39.194508 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.194532 (+ 24us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:39.194705 (+ 173us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:39.194707 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:41:39.194758 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:41:39.194798 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":123,"threads_started":1}
I20260926 05:41:39.201984 25523 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:39.202107 25523 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56238:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:39.202710 25523 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 94d84d5071224e86b8e6a87aa9671516: Sending DeleteTablet for 3 replicas of tablet 38ae18d064104d7fb91f14a25f7c7753
I20260926 05:41:39.202976 25660 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae18d064104d7fb91f14a25f7c7753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:39 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:38348
I20260926 05:41:39.202966 25940 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae18d064104d7fb91f14a25f7c7753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:39 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37550
I20260926 05:41:39.202977 25800 tablet_service.cc:1514] Processing DeleteTablet for tablet 38ae18d064104d7fb91f14a25f7c7753 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:39 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:34344
I20260926 05:41:39.203059 26050 tablet_replica.cc:330] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: stopping tablet replica
I20260926 05:41:39.203073 26051 tablet_replica.cc:330] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: stopping tablet replica
I20260926 05:41:39.203087 26052 tablet_replica.cc:330] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: stopping tablet replica
I20260926 05:41:39.203105 26050 raft_consensus.cc:2229] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:41:39.203136 26052 raft_consensus.cc:2229] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:39.203143 26051 raft_consensus.cc:2229] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:39.203168 26052 raft_consensus.cc:2258] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:39.203181 26050 raft_consensus.cc:2258] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:39.203184 26051 raft_consensus.cc:2258] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:39.203461 26050 ts_tablet_manager.cc:1905] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:39.203622 26052 ts_tablet_manager.cc:1905] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:39.204011 26051 ts_tablet_manager.cc:1905] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:39.205104 26050 ts_tablet_manager.cc:1918] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:39.205169 26050 log.cc:1196] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/38ae18d064104d7fb91f14a25f7c7753
I20260926 05:41:39.205385 26050 ts_tablet_manager.cc:1939] T 38ae18d064104d7fb91f14a25f7c7753 P f521b60381af419691b270be272f1433: Deleting consensus metadata
I20260926 05:41:39.205413 26052 ts_tablet_manager.cc:1918] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:39.205470 26052 log.cc:1196] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/38ae18d064104d7fb91f14a25f7c7753
I20260926 05:41:39.205725 26052 ts_tablet_manager.cc:1939] T 38ae18d064104d7fb91f14a25f7c7753 P ea68056345cf46c994f5e10fdb9591c9: Deleting consensus metadata
I20260926 05:41:39.205724 26051 ts_tablet_manager.cc:1918] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:39.205803 26051 log.cc:1196] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/38ae18d064104d7fb91f14a25f7c7753
I20260926 05:41:39.206025 26051 ts_tablet_manager.cc:1939] T 38ae18d064104d7fb91f14a25f7c7753 P 6efea551249b493ebeebd1c944328fad: Deleting consensus metadata
I20260926 05:41:39.206096 25507 catalog_manager.cc:4841] TS ea68056345cf46c994f5e10fdb9591c9 (127.23.213.194:45033): tablet 38ae18d064104d7fb91f14a25f7c7753 (table test-table [id=38e6defd595e40488d5b122e27ac298c]) successfully deleted
I20260926 05:41:39.206295 25510 catalog_manager.cc:4841] TS f521b60381af419691b270be272f1433 (127.23.213.195:33769): tablet 38ae18d064104d7fb91f14a25f7c7753 (table test-table [id=38e6defd595e40488d5b122e27ac298c]) successfully deleted
I20260926 05:41:39.206338 25507 catalog_manager.cc:4841] TS 6efea551249b493ebeebd1c944328fad (127.23.213.193:35259): tablet 38ae18d064104d7fb91f14a25f7c7753 (table test-table [id=38e6defd595e40488d5b122e27ac298c]) successfully deleted
Sep 26 05:41:39 dist-test-slave-1fdt krb5kdc[25478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401298, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:39.214615 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.206139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38378 (local address 127.23.213.193:35259)
0926 05:41:39.206200 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.206203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.206501 (+ 298us) server_negotiation.cc:407] Connection header received
0926 05:41:39.206574 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.206574 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.206584 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.206628 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:39.207150 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.207626 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.208339 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.208605 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.208698 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.211646 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:39.211659 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:39.211659 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:39.211667 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:39.213339 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:39.213711 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:39.213714 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:39.213715 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:39.213823 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:39.214028 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:39.214029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:39.214029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:39.214179 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:39.214363 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.214427 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.214510 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 05:41:39.214931 25720 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38378
I20260926 05:41:39.222384 25572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:39.215878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56264 (local address 127.23.213.254:40053)
0926 05:41:39.215925 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:39.215927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:39.216113 (+ 186us) server_negotiation.cc:407] Connection header received
0926 05:41:39.216221 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:39.216222 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:39.216234 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:39.216316 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:39.216796 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.217365 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.218158 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:39.218370 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:39.218456 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:39.219641 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:39.219642 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:39.219642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:39.219650 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:39.221058 (+ 1408us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:39.221419 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:39.221420 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:39.221421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:39.221483 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:39.221700 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:39.221701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:39.221702 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:39.221862 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:39.221959 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:39.222144 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 05:41:39.222225 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:41:39.222594 25523 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56264
I20260926 05:41:39.223244 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 25595
I20260926 05:41:39.230568 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 25735
I20260926 05:41:39.237298 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 25875
I20260926 05:41:39.243620 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 25491
2026-09-26T05:41:39Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2271 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:39 dist-test-slave-1fdt krb5kdc[26093](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:39 dist-test-slave-1fdt krb5kdc[26093](info): set up 2 sockets
Sep 26 05:41:39 dist-test-slave-1fdt krb5kdc[26093](info): commencing operation
krb5kdc: starting...
W20260926 05:41:40.293529 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:40.408499 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41699
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:41699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:40.489796 26106 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:40.490028 26106 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:40.490051 26106 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:40.491237 26106 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:40.491278 26106 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:41:40.491291 26106 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:40.491333 26106 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:40.491365 26106 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:40.491384 26106 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:40.492883 26106 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:41699
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26106
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:40.493155 26106 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:40.493394 26106 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:40.498566 26111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:40.498544 26106 server_base.cc:1034] running on GCE node
W20260926 05:41:40.498526 26112 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:40.498529 26114 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:40.499030 26106 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:40.499310 26106 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:40.500459 26106 hybrid_clock.cc:648] HybridClock initialized: now 1790401300500440 us; error 37 us; skew 500 ppm
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:40.503131 26106 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:40.503580 26106 webserver.cc:460] Webserver started at http://127.23.213.254:36979/ using document root <none> and password file <none>
I20260926 05:41:40.503820 26106 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:40.503892 26106 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:40.504045 26106 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:40.505119 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "c5632a45822347f084b440ca1cbf89b0"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "d48a345ea4391c706625ea51c73791a4"
server_key_iv: "55d4e1ca97aa230d2e3d73af04af981a"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.505486 26106 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "c5632a45822347f084b440ca1cbf89b0"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "d48a345ea4391c706625ea51c73791a4"
server_key_iv: "55d4e1ca97aa230d2e3d73af04af981a"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.506938 26106 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:40.507686 26121 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:40.507855 26106 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:40.507965 26106 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "c5632a45822347f084b440ca1cbf89b0"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "d48a345ea4391c706625ea51c73791a4"
server_key_iv: "55d4e1ca97aa230d2e3d73af04af981a"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.508057 26106 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:40.520412 26106 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:40.522833 26106 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:40.523005 26106 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:40.527727 26106 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:41699
I20260926 05:41:40.527776 26183 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:41699 every 8 connection(s)
I20260926 05:41:40.528107 26106 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:40.528697 26184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:40.531032 26184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Bootstrap starting.
I20260926 05:41:40.531613 26184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:40.531899 26184 log.cc:824] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:40.532558 26184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: No bootstrap required, opened a new log
I20260926 05:41:40.533475 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 26106
I20260926 05:41:40.533547 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:40.533727 24407 external_mini_cluster.cc:1428] Setting key fea01e748e13365a4c0fc07bed1dbb8e
I20260926 05:41:40.534994 26184 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5632a45822347f084b440ca1cbf89b0" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41699 } }
I20260926 05:41:40.535159 26184 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:40.535203 26184 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5632a45822347f084b440ca1cbf89b0, State: Initialized, Role: FOLLOWER
I20260926 05:41:40.535338 26184 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5632a45822347f084b440ca1cbf89b0" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41699 } }
I20260926 05:41:40.535424 26184 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:40.535465 26184 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:40.535517 26184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:40.536046 26184 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5632a45822347f084b440ca1cbf89b0" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41699 } }
I20260926 05:41:40.536170 26184 leader_election.cc:302] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5632a45822347f084b440ca1cbf89b0; no voters:
I20260926 05:41:40.536370 26184 leader_election.cc:288] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:40.536403 26189 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:40.536504 26189 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [term 1 LEADER]: Becoming Leader. State: Replica: c5632a45822347f084b440ca1cbf89b0, State: Running, Role: LEADER
I20260926 05:41:40.536585 26189 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5632a45822347f084b440ca1cbf89b0" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41699 } }
I20260926 05:41:40.536643 26184 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:40.536919 26191 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5632a45822347f084b440ca1cbf89b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5632a45822347f084b440ca1cbf89b0" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41699 } } }
I20260926 05:41:40.537007 26191 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:40.537242 26190 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5632a45822347f084b440ca1cbf89b0. Latest consensus state: current_term: 1 leader_uuid: "c5632a45822347f084b440ca1cbf89b0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5632a45822347f084b440ca1cbf89b0" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41699 } } }
I20260926 05:41:40.537367 26190 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:40.537524 26194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:40.538069 26194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:40.539852 26194 catalog_manager.cc:1347] Generated new cluster ID: c7244a08c8794e7f87921e6b20563402
I20260926 05:41:40.539904 26194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:40.545493 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.534316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52402 (local address 127.23.213.254:41699)
0926 05:41:40.534522 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:40.534530 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:40.534554 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:41:40.534981 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:40.534987 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:40.535031 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:40.535191 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:40.535897 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.537183 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.538311 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.538930 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.539037 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.541803 (+ 2766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:40.541807 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:40.541807 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:40.541819 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:40.543816 (+ 1997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.544231 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.544234 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.544234 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.544351 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.544566 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.544567 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.544567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.544865 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:40.545015 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:40.545150 (+ 135us) server_negotiation.cc:299] Negotiation successful
0926 05:41:40.545291 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260926 05:41:40.548455 26194 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:40.549170 26194 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:40.556140 26194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Generated new TSK 0
I20260926 05:41:40.556370 26194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:40.582803 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41699
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:40.666097 26210 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:40.666311 26210 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:40.666337 26210 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:40.667513 26210 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:40.667552 26210 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:40.667573 26210 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:41:40.667585 26210 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:41:40.667603 26210 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:40.669085 26210 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26210
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:40.669271 26210 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:40.669452 26210 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:41:40.674471 26210 server_base.cc:1034] running on GCE node
W20260926 05:41:40.674445 26218 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:40.674434 26215 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:40.674439 26216 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:40.674799 26210 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:40.675058 26210 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:40.676213 26210 hybrid_clock.cc:648] HybridClock initialized: now 1790401300676201 us; error 28 us; skew 500 ppm
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:40.678750 26210 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:40.679184 26210 webserver.cc:460] Webserver started at http://127.23.213.193:33359/ using document root <none> and password file <none>
I20260926 05:41:40.679428 26210 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:40.679476 26210 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:40.679641 26210 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:40.680718 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "d133b12692894037a711baaf6de5e1a9"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "ddc8e84438b50c03cf418f21999d2a36"
server_key_iv: "1b6c2b2c9f66e544f1504f92e3835135"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.681078 26210 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "d133b12692894037a711baaf6de5e1a9"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "ddc8e84438b50c03cf418f21999d2a36"
server_key_iv: "1b6c2b2c9f66e544f1504f92e3835135"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.682482 26210 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:40.683274 26225 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:40.683473 26210 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:40.683595 26210 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "d133b12692894037a711baaf6de5e1a9"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "ddc8e84438b50c03cf418f21999d2a36"
server_key_iv: "1b6c2b2c9f66e544f1504f92e3835135"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.683693 26210 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:40.694689 26210 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:40.696563 26210 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:40.696729 26210 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:40.697250 26210 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:40.697299 26210 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:40.697325 26210 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:40.697379 26210 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:40.705336 26210 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:41101
I20260926 05:41:40.705386 26340 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:41101 every 8 connection(s)
I20260926 05:41:40.705838 26210 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:40.707659 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 26210
I20260926 05:41:40.707762 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:40.707947 24407 external_mini_cluster.cc:1428] Setting key f7e2c26e129f2629e56ba50bb3b7001c
I20260926 05:41:40.708098 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.697782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:43997 (local address 127.23.213.254:41699)
0926 05:41:40.697858 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:40.697861 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:40.698060 (+ 199us) server_negotiation.cc:407] Connection header received
0926 05:41:40.698539 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:40.698540 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:40.698562 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:40.698654 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:40.699210 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.700085 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.701683 (+ 1598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.702087 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.702179 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.704833 (+ 2654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:40.704834 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:40.704835 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:40.704843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:40.706664 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.707025 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.707027 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.707027 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.707084 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.707296 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.707297 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.707297 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.707406 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:40.707467 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:40.707868 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 05:41:40.707962 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 05:41:40.708233 26234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.697731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41699 (local address 127.23.213.193:43997)
0926 05:41:40.697926 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:40.697967 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:40.698441 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:40.698720 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:40.698723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:40.698950 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:40.699132 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.699136 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.700204 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.700205 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.701579 (+ 1374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.701583 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.702193 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.702194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.702330 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.702884 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:40.702893 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:40.704736 (+ 1843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:40.706737 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.706741 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.706743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.706948 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.707148 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.707148 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.707148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.707239 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.707488 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:40.707494 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:40.707671 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 05:41:40.707800 (+ 129us) client_negotiation.cc:240] Negotiation successful
0926 05:41:40.707972 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":128,"threads_started":1}
I20260926 05:41:40.708796 26341 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41699
I20260926 05:41:40.708879 26341 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:40.709062 26341 heartbeater.cc:511] Master 127.23.213.254:41699 requested a full tablet report, sending...
I20260926 05:41:40.710306 26137 ts_manager.cc:194] Registered new tserver with Master: d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101)
I20260926 05:41:40.711012 26137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:43997
I20260926 05:41:40.716030 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.710098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:57985 (local address 127.23.213.254:41699)
0926 05:41:40.710140 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:40.710143 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:40.710308 (+ 165us) server_negotiation.cc:407] Connection header received
0926 05:41:40.710329 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:40.710330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:40.710338 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:40.710405 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:40.710749 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.711357 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.712180 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.712389 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.712475 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.713392 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:40.713403 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:40.713404 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:40.713410 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:40.714861 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.715225 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.715226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.715227 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.715286 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.715502 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.715503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.715504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.715610 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:40.715701 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:40.715864 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 05:41:40.715942 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:41:40.716046 26347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.709893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41699 (local address 127.23.213.193:57985)
0926 05:41:40.710215 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:40.710226 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:40.710289 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:40.710426 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:40.710427 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:40.710588 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:40.710665 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.710667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.711451 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.711452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.712083 (+ 631us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.712085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.712477 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.712478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.712550 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.712984 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:40.712988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:40.713283 (+ 295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:40.714956 (+ 1673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.714957 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.714958 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.715130 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.715353 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.715354 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.715355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.715412 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.715695 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:40.715696 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:40.715739 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 05:41:40.715839 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 05:41:40.715942 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":4,"thread_start_us":77,"threads_started":1}
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:40.734483 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41699
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:40.817369 26350 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:40.817624 26350 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:40.817665 26350 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:40.818918 26350 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:40.819006 26350 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:40.819046 26350 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:41:40.819103 26350 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:41:40.819144 26350 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:40.820658 26350 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26350
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:40.820957 26350 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:40.821198 26350 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:40.826253 26358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:40.826339 26355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:40.826253 26356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:40.826426 26350 server_base.cc:1034] running on GCE node
I20260926 05:41:40.826609 26350 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:40.826815 26350 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:40.827952 26350 hybrid_clock.cc:648] HybridClock initialized: now 1790401300827926 us; error 42 us; skew 500 ppm
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:40.830495 26350 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:40.830979 26350 webserver.cc:460] Webserver started at http://127.23.213.194:39855/ using document root <none> and password file <none>
I20260926 05:41:40.831212 26350 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:40.831257 26350 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:40.831423 26350 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:40.832752 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "99f9c59136cf41efb2a49eeea5ad937e"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "c56bbb4dc1a1603d25f80a3fc7bd1266"
server_key_iv: "06e6dfcb1a87d7eea90102fa3a151a1e"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.833235 26350 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "99f9c59136cf41efb2a49eeea5ad937e"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "c56bbb4dc1a1603d25f80a3fc7bd1266"
server_key_iv: "06e6dfcb1a87d7eea90102fa3a151a1e"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.834739 26350 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 05:41:40.835547 26365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:40.835738 26350 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:41:40.835834 26350 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "99f9c59136cf41efb2a49eeea5ad937e"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "c56bbb4dc1a1603d25f80a3fc7bd1266"
server_key_iv: "06e6dfcb1a87d7eea90102fa3a151a1e"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.835914 26350 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:40.847877 26350 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:40.850456 26350 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:40.850638 26350 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:40.851239 26350 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:40.851284 26350 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:40.851310 26350 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:40.851325 26350 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:40.859269 26350 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:45973
I20260926 05:41:40.859341 26480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:45973 every 8 connection(s)
I20260926 05:41:40.859716 26350 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:41:40.861506 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.852000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:46431 (local address 127.23.213.254:41699)
0926 05:41:40.852072 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:40.852075 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:40.852086 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:41:40.852518 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:40.852520 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:40.852532 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:40.852579 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:40.853490 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.854326 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.855502 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.855718 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.855807 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.858447 (+ 2640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:40.858449 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:40.858449 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:40.858458 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:40.860196 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.860704 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.860706 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.860706 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.860766 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.860957 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.860963 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.860964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.861082 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:40.861133 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:40.861377 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 05:41:40.861442 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:41:40.861615 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.851635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41699 (local address 127.23.213.194:46431)
0926 05:41:40.851808 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:40.851854 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:40.852408 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:40.852668 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:40.852672 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:40.853054 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:40.853380 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.853387 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.854432 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.854433 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.855390 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.855393 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.855818 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.855819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.855953 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.856459 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:40.856467 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:40.858329 (+ 1862us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:40.860393 (+ 2064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.860396 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.860398 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.860620 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.860811 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.860812 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.860812 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.860899 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.861124 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:40.861126 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:40.861241 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 05:41:40.861327 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:41:40.861461 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260926 05:41:40.862202 26481 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41699
I20260926 05:41:40.862294 26481 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:40.862522 26481 heartbeater.cc:511] Master 127.23.213.254:41699 requested a full tablet report, sending...
I20260926 05:41:40.862946 26137 ts_manager.cc:194] Registered new tserver with Master: 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973)
I20260926 05:41:40.863344 26137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:46431
I20260926 05:41:40.868877 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.863322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:56453 (local address 127.23.213.254:41699)
0926 05:41:40.863360 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:40.863362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:40.863477 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:41:40.863563 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:40.863563 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:40.863574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:40.863643 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:40.863976 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.864514 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.865331 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.865520 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.865580 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.866545 (+ 965us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:40.866546 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:40.866546 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:40.866552 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:40.867930 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.868250 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.868250 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.868251 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.868297 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:40.868482 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:40.868482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:40.868483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:40.868601 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:40.868649 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:40.868767 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 05:41:40.868820 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:41:40.869037 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:40.863274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41699 (local address 127.23.213.194:56453)
0926 05:41:40.863418 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:40.863428 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:40.863493 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:40.863650 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:40.863651 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:40.863842 (+ 191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:40.863899 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.863900 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.864601 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.864601 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.865177 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:40.865179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:40.865609 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:40.865610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:40.865686 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:40.866154 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:40.866167 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:40.866467 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:40.868011 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.868012 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.868013 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.868183 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.868354 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:40.868354 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:40.868355 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:40.868407 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:40.868653 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:40.868654 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:40.868690 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 05:41:40.868751 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:41:40.868833 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260926 05:41:40.869473 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 26350
I20260926 05:41:40.869566 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:40.869787 24407 external_mini_cluster.cc:1428] Setting key ef419167eb8b4a170fd22015ed97384c
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:40.894532 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41699
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:40.977178 26490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:40.977409 26490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:40.977444 26490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:40.978683 26490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:40.978729 26490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:40.978752 26490 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 05:41:40.978803 26490 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:41:40.978843 26490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:40.980368 26490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:41699
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:40.980654 26490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:40.980897 26490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:40.985953 26495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:40.986047 26496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:40.986047 26498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:40.986335 26490 server_base.cc:1034] running on GCE node
I20260926 05:41:40.986511 26490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:40.986704 26490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:40.987840 26490 hybrid_clock.cc:648] HybridClock initialized: now 1790401300987829 us; error 27 us; skew 500 ppm
Sep 26 05:41:40 dist-test-slave-1fdt krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:40.990398 26490 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:40.990871 26490 webserver.cc:460] Webserver started at http://127.23.213.195:38709/ using document root <none> and password file <none>
I20260926 05:41:40.991114 26490 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:40.991192 26490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:40.991340 26490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:40.992424 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "813557481cb140f2af611bd9739b8e1b"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "d71cfbf37e5a7d30a073ab5fe8b1570d"
server_key_iv: "5f926364e14ed0651a791e5183749cd1"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.992792 26490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "813557481cb140f2af611bd9739b8e1b"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "d71cfbf37e5a7d30a073ab5fe8b1570d"
server_key_iv: "5f926364e14ed0651a791e5183749cd1"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.994257 26490 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:40.995074 26505 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:40.995270 26490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:40.995375 26490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "813557481cb140f2af611bd9739b8e1b"
format_stamp: "Formatted at 2026-09-26 05:41:40 on dist-test-slave-1fdt"
server_key: "d71cfbf37e5a7d30a073ab5fe8b1570d"
server_key_iv: "5f926364e14ed0651a791e5183749cd1"
server_key_version: "encryptionkey@0"
I20260926 05:41:40.995450 26490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:41.014185 26490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:41.016218 26490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:41.016390 26490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:41.016911 26490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:41.016976 26490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:41.017036 26490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:41.017066 26490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:41:41 dist-test-slave-1fdt krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:41.024499 26490 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:37967
I20260926 05:41:41.024567 26620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:37967 every 8 connection(s)
I20260926 05:41:41.024920 26490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:41.027442 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.017472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:59577 (local address 127.23.213.254:41699)
0926 05:41:41.017561 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.017564 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.017770 (+ 206us) server_negotiation.cc:407] Connection header received
0926 05:41:41.018293 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.018295 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.018309 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.018386 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:41.019246 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.020101 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.021133 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.021500 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.021580 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.024733 (+ 3153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:41.024734 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:41.024735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:41.024743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:41.026336 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.026700 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.026701 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.026701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.026752 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.026949 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.026949 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.026950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.027045 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:41.027092 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.027319 (+ 227us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.027383 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:41:41.027693 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.017436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41699 (local address 127.23.213.195:59577)
0926 05:41:41.017617 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.017664 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.018192 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.018480 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.018491 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.018848 (+ 357us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:41.019134 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.019140 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.020210 (+ 1070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.020211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.021026 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.021029 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.021680 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.021682 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.021908 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.022508 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:41.022518 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:41.024607 (+ 2089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:41.026419 (+ 1812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:41.026423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:41.026425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:41.026626 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:41.026806 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:41.026806 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:41.026806 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:41.026898 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:41.027096 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:41.027100 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:41.027216 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.027296 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.027409 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":115,"threads_started":1}
I20260926 05:41:41.028242 26621 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41699
I20260926 05:41:41.028316 26621 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:41.028512 26621 heartbeater.cc:511] Master 127.23.213.254:41699 requested a full tablet report, sending...
I20260926 05:41:41.028867 26137 ts_manager.cc:194] Registered new tserver with Master: 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:41.029296 26137 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:59577
I20260926 05:41:41.029919 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 26490
I20260926 05:41:41.030014 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:41.030192 24407 external_mini_cluster.cc:1428] Setting key fd36d1d95470571a8a598175c29b7d27
I20260926 05:41:41.031129 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:41:41.035231 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.029309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:55319 (local address 127.23.213.254:41699)
0926 05:41:41.029350 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.029352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.029426 (+ 74us) server_negotiation.cc:407] Connection header received
0926 05:41:41.029486 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.029487 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.029496 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.029545 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:41.029930 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.030406 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.031296 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.031559 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.031638 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.032507 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:41.032508 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:41.032508 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:41.032515 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:41.034160 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.034477 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.034478 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.034479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.034530 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.034738 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.034739 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.034739 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.034847 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:41.034922 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.035066 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.035144 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:41:41.035251 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.029221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41699 (local address 127.23.213.195:55319)
0926 05:41:41.029364 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.029376 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.029439 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.029550 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.029551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.029733 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:41.029846 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.029848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.030508 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.030508 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.031232 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.031234 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.031647 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.031648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.031694 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.032128 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:41.032132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:41.032404 (+ 272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:41.034225 (+ 1821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:41.034226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:41.034226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:41.034392 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:41.034605 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:41.034606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:41.034606 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:41.034669 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:41.034914 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:41.034915 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:41.034958 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.035041 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.035140 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":84,"threads_started":1}
Sep 26 05:41:41 dist-test-slave-1fdt krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:41:41 dist-test-slave-1fdt krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401301, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:41.053603 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.047309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52412 (local address 127.23.213.254:41699)
0926 05:41:41.047365 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.047367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.047377 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:41:41.047437 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.047437 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.047445 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.047493 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:41.047991 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.048459 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.049121 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.049304 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.049353 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.051444 (+ 2091us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:41.051445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:41.051445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:41.051451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:41.052680 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.053011 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.053012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.053012 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.053062 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.053228 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.053229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.053229 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.053324 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:41.053396 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.053477 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.053540 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:41:41.055187 26137 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:41:41.056960 26137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:41.063349 26640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.059300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41101 (local address 127.0.0.1:43276)
0926 05:41:41.059763 (+ 463us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.059771 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.059808 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.060156 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.060157 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.060159 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.060216 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.060217 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.061042 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.061043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.062332 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.062335 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.063041 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.063042 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.063119 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.063132 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.063213 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.063268 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":86,"threads_started":1}
I20260926 05:41:41.063540 26638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.059286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:37967 (local address 127.0.0.1:47920)
0926 05:41:41.059436 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.059476 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.059541 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.059890 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.059892 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.059894 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.059972 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.059974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.061276 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.061277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.062317 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.062320 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.063309 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.063310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.063369 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.063373 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.063416 (+ 43us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.063459 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":47,"threads_started":1}
I20260926 05:41:41.063546 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.059643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43276 (local address 127.23.213.193:41101)
0926 05:41:41.059819 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.059822 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.059831 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:41:41.059985 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.059987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.060002 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.060076 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.060285 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.060946 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.062433 (+ 1487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.062938 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.063041 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.063101 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.063393 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.063437 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260926 05:41:41.063552 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.059446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37354 (local address 127.23.213.194:45973)
0926 05:41:41.059609 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.059614 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.059735 (+ 121us) server_negotiation.cc:407] Connection header received
0926 05:41:41.059796 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.059798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.059817 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.059888 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.060058 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.061171 (+ 1113us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.062438 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.063113 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.063198 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.063294 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.063320 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.063357 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260926 05:41:41.063839 26639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.059286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45973 (local address 127.0.0.1:37354)
0926 05:41:41.059622 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.059631 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.059698 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.059954 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.059954 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.059956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.059984 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.059986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.061256 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.061257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.062379 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.062382 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.063202 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.063202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.063255 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.063258 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.063297 (+ 39us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.063328 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260926 05:41:41.063925 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.059457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47920 (local address 127.23.213.195:37967)
0926 05:41:41.059696 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.059702 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.059712 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:41:41.059754 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.059755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.059773 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.059864 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.060058 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.061188 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.062398 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.063113 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.063196 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.063716 (+ 520us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.063748 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.063793 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":197,"thread_start_us":84,"threads_started":1}
I20260926 05:41:41.064241 26275 tablet_service.cc:1467] Processing CreateTablet for tablet b7e0d9ff0d24434e884e1741d83e7e4d (DEFAULT_TABLE table=test-table [id=4368fc62e42e4cac9dae891b78fff4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:41.064504 26555 tablet_service.cc:1467] Processing CreateTablet for tablet b7e0d9ff0d24434e884e1741d83e7e4d (DEFAULT_TABLE table=test-table [id=4368fc62e42e4cac9dae891b78fff4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:41.064611 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e0d9ff0d24434e884e1741d83e7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.064833 26555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e0d9ff0d24434e884e1741d83e7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.064836 26415 tablet_service.cc:1467] Processing CreateTablet for tablet b7e0d9ff0d24434e884e1741d83e7e4d (DEFAULT_TABLE table=test-table [id=4368fc62e42e4cac9dae891b78fff4db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:41.065171 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b7e0d9ff0d24434e884e1741d83e7e4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.067970 26644 tablet_bootstrap.cc:492] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Bootstrap starting.
I20260926 05:41:41.068306 26645 tablet_bootstrap.cc:492] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Bootstrap starting.
I20260926 05:41:41.068670 26644 tablet_bootstrap.cc:654] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.068971 26644 log.cc:824] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:41.069146 26645 tablet_bootstrap.cc:654] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.069439 26645 log.cc:824] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:41.069720 26644 tablet_bootstrap.cc:492] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: No bootstrap required, opened a new log
I20260926 05:41:41.069823 26644 ts_tablet_manager.cc:1397] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:41.069818 26646 tablet_bootstrap.cc:492] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Bootstrap starting.
I20260926 05:41:41.070308 26646 tablet_bootstrap.cc:654] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.070590 26646 log.cc:824] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:41.071137 26645 tablet_bootstrap.cc:492] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: No bootstrap required, opened a new log
I20260926 05:41:41.071200 26645 ts_tablet_manager.cc:1397] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:41:41.071462 26644 raft_consensus.cc:348] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.071630 26644 raft_consensus.cc:374] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.071681 26644 raft_consensus.cc:729] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f9c59136cf41efb2a49eeea5ad937e, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.071789 26644 consensus_queue.cc:260] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.072050 26644 ts_tablet_manager.cc:1428] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:41.072109 26481 heartbeater.cc:503] Master 127.23.213.254:41699 was elected leader, sending a full tablet report...
I20260926 05:41:41.073179 26645 raft_consensus.cc:348] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.073355 26645 raft_consensus.cc:374] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.073400 26645 raft_consensus.cc:729] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813557481cb140f2af611bd9739b8e1b, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.073516 26645 consensus_queue.cc:260] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.073782 26646 tablet_bootstrap.cc:492] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: No bootstrap required, opened a new log
I20260926 05:41:41.073804 26645 ts_tablet_manager.cc:1428] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:41:41.073851 26646 ts_tablet_manager.cc:1397] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 05:41:41.073882 26621 heartbeater.cc:503] Master 127.23.213.254:41699 was elected leader, sending a full tablet report...
I20260926 05:41:41.075466 26646 raft_consensus.cc:348] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.075583 26646 raft_consensus.cc:374] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.075604 26646 raft_consensus.cc:729] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.075683 26646 consensus_queue.cc:260] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.075887 26646 ts_tablet_manager.cc:1428] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:41.075922 26341 heartbeater.cc:503] Master 127.23.213.254:41699 was elected leader, sending a full tablet report...
I20260926 05:41:41.094892 26651 raft_consensus.cc:482] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:41.095008 26651 raft_consensus.cc:504] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.095415 26651 leader_election.cc:288] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101)
I20260926 05:41:41.098362 26514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.095575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45973 (local address 127.23.213.195:59755)
0926 05:41:41.095634 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.095645 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.095684 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.095862 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.095863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.095864 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.095934 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.095936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.096665 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.096666 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.097442 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.097444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.098111 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.098112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.098196 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.098200 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.098256 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.098299 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 05:41:41.098393 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.095711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:59755 (local address 127.23.213.194:45973)
0926 05:41:41.095755 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.095756 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.095764 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:41:41.095790 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.095791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.095800 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.095846 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.096035 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.096581 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.097540 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.098038 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.098095 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.098102 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.098271 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.098310 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:41:41.098534 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.095703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:54995 (local address 127.23.213.193:41101)
0926 05:41:41.095747 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.095749 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.095847 (+ 98us) server_negotiation.cc:407] Connection header received
0926 05:41:41.095935 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.095936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.095947 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.096003 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.096193 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.096679 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.097603 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.098179 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.098255 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.098264 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.098430 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.098474 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:41:41.098730 26653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.095621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41101 (local address 127.23.213.195:54995)
0926 05:41:41.095786 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.095795 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.095873 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.096034 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.096035 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.096037 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.096110 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.096112 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.096764 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.096765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.097458 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.097460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.098275 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.098276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.098344 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.098347 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.098405 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.098630 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":1,"thread_start_us":103,"threads_started":1}
I20260926 05:41:41.098806 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e0d9ff0d24434e884e1741d83e7e4d" candidate_uuid: "813557481cb140f2af611bd9739b8e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e" is_pre_election: true
I20260926 05:41:41.098973 26435 raft_consensus.cc:2454] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813557481cb140f2af611bd9739b8e1b in term 0.
I20260926 05:41:41.099104 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e0d9ff0d24434e884e1741d83e7e4d" candidate_uuid: "813557481cb140f2af611bd9739b8e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d133b12692894037a711baaf6de5e1a9" is_pre_election: true
I20260926 05:41:41.099195 26506 leader_election.cc:302] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813557481cb140f2af611bd9739b8e1b, 99f9c59136cf41efb2a49eeea5ad937e; no voters:
I20260926 05:41:41.099275 26295 raft_consensus.cc:2454] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 813557481cb140f2af611bd9739b8e1b in term 0.
I20260926 05:41:41.099345 26651 raft_consensus.cc:2781] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:41.099412 26651 raft_consensus.cc:482] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:41.099438 26651 raft_consensus.cc:3037] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.100116 26651 raft_consensus.cc:504] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.100267 26651 leader_election.cc:288] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [CANDIDATE]: Term 1 election: Requested vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101)
I20260926 05:41:41.100421 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e0d9ff0d24434e884e1741d83e7e4d" candidate_uuid: "813557481cb140f2af611bd9739b8e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e"
I20260926 05:41:41.100481 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b7e0d9ff0d24434e884e1741d83e7e4d" candidate_uuid: "813557481cb140f2af611bd9739b8e1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d133b12692894037a711baaf6de5e1a9"
I20260926 05:41:41.100512 26435 raft_consensus.cc:3037] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.100584 26295 raft_consensus.cc:3037] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.101217 26435 raft_consensus.cc:2454] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813557481cb140f2af611bd9739b8e1b in term 1.
I20260926 05:41:41.101261 26295 raft_consensus.cc:2454] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 813557481cb140f2af611bd9739b8e1b in term 1.
I20260926 05:41:41.101429 26506 leader_election.cc:302] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813557481cb140f2af611bd9739b8e1b, 99f9c59136cf41efb2a49eeea5ad937e; no voters:
I20260926 05:41:41.101552 26651 raft_consensus.cc:2781] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:41.101697 26651 raft_consensus.cc:686] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 1 LEADER]: Becoming Leader. State: Replica: 813557481cb140f2af611bd9739b8e1b, State: Running, Role: LEADER
I20260926 05:41:41.101825 26651 consensus_queue.cc:237] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } }
I20260926 05:41:41.102598 26137 catalog_manager.cc:5495] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b reported cstate change: term changed from 0 to 1, leader changed from <none> to 813557481cb140f2af611bd9739b8e1b (127.23.213.195). New cstate: current_term: 1 leader_uuid: "813557481cb140f2af611bd9739b8e1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } health_report { overall_health: UNKNOWN } } }
W20260926 05:41:41.110280 26482 tablet.cc:2369] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 05:41:41 dist-test-slave-1fdt krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401300, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:41.137521 26188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.130603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52416 (local address 127.23.213.254:41699)
0926 05:41:41.130683 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.130685 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.130825 (+ 140us) server_negotiation.cc:407] Connection header received
0926 05:41:41.130878 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.130878 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.130887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.130950 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:41.131453 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.131949 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.132650 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.132835 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.132886 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.135011 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:41.135012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:41.135012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:41.135018 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:41.136301 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.136668 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.136669 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.136670 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.136718 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:41.136906 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:41.136907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:41.136907 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:41.137045 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:41.137107 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.137276 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.137383 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:41:41.137617 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.130529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41699 (local address 127.0.0.1:52416)
0926 05:41:41.130735 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.130748 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.130826 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.130973 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.130974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.131304 (+ 330us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:41.131365 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.131368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.132023 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.132024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.132574 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.132576 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.132919 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.132920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.132975 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.133481 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:41.133486 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:41.134926 (+ 1440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:41.136370 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:41.136376 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:41.136378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:41.136595 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:41.136779 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:41.136779 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:41.136780 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:41.136842 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:41.137106 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:41.137115 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:41.137159 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.137253 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.137447 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"mutex_wait_us":95,"thread_start_us":132,"threads_started":1}
I20260926 05:41:41.139950 26137 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:52416:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:41:41.140209 26137 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:41.141769 26415 tablet_service.cc:1467] Processing CreateTablet for tablet dfcbaa055b064e73aa79fbd606d39601 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7c194a16d7f466f918b20826116abb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:41:41.141769 26275 tablet_service.cc:1467] Processing CreateTablet for tablet dfcbaa055b064e73aa79fbd606d39601 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7c194a16d7f466f918b20826116abb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:41:41.141821 26555 tablet_service.cc:1467] Processing CreateTablet for tablet dfcbaa055b064e73aa79fbd606d39601 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=e7c194a16d7f466f918b20826116abb2]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:41:41.141942 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcbaa055b064e73aa79fbd606d39601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.141942 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcbaa055b064e73aa79fbd606d39601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.142012 26555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dfcbaa055b064e73aa79fbd606d39601. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.143446 26646 tablet_bootstrap.cc:492] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: Bootstrap starting.
I20260926 05:41:41.143563 26645 tablet_bootstrap.cc:492] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: Bootstrap starting.
I20260926 05:41:41.144038 26645 tablet_bootstrap.cc:654] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.144138 26646 tablet_bootstrap.cc:654] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.144418 26644 tablet_bootstrap.cc:492] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: Bootstrap starting.
I20260926 05:41:41.144776 26645 tablet_bootstrap.cc:492] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: No bootstrap required, opened a new log
I20260926 05:41:41.144826 26645 ts_tablet_manager.cc:1397] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:41.144986 26645 raft_consensus.cc:348] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.145074 26645 raft_consensus.cc:374] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.145071 26644 tablet_bootstrap.cc:654] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.145097 26645 raft_consensus.cc:729] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813557481cb140f2af611bd9739b8e1b, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.145154 26645 consensus_queue.cc:260] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.145294 26645 ts_tablet_manager.cc:1428] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:41.145357 26651 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.145469 26651 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:41.145516 26646 tablet_bootstrap.cc:492] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: No bootstrap required, opened a new log
I20260926 05:41:41.145555 26646 ts_tablet_manager.cc:1397] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:41.145709 26646 raft_consensus.cc:348] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.145825 26646 raft_consensus.cc:374] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.145852 26646 raft_consensus.cc:729] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.145910 26646 consensus_queue.cc:260] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.145978 26644 tablet_bootstrap.cc:492] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: No bootstrap required, opened a new log
I20260926 05:41:41.146008 26646 ts_tablet_manager.cc:1428] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:41.146015 26644 ts_tablet_manager.cc:1397] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:41.146075 26652 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.146173 26652 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:41.146168 26644 raft_consensus.cc:348] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.146267 26644 raft_consensus.cc:374] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.146293 26644 raft_consensus.cc:729] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f9c59136cf41efb2a49eeea5ad937e, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.146338 26644 consensus_queue.cc:260] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.146436 26644 ts_tablet_manager.cc:1428] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:41.146492 26650 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.146633 26650 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:41.171751 26650 raft_consensus.cc:482] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:41.171838 26650 raft_consensus.cc:504] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.172202 26650 leader_election.cc:288] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:41.175478 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.172449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:60883 (local address 127.23.213.195:37967)
0926 05:41:41.172517 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.172519 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.172532 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:41:41.172561 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.172562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.172570 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.172639 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.172857 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.173642 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.174729 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.175094 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.175195 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.175205 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.175362 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.175403 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:41:41.175488 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.172499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:47657 (local address 127.23.213.193:41101)
0926 05:41:41.172541 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.172543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.172703 (+ 160us) server_negotiation.cc:407] Connection header received
0926 05:41:41.172737 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.172738 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.172746 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.172788 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.172945 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.173641 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.174716 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.175095 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.175198 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.175207 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.175379 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.175419 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:41:41.175501 26374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.172387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:37967 (local address 127.23.213.194:60883)
0926 05:41:41.172440 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.172452 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.172495 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.172677 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.172679 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.172681 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.172760 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.172762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.173728 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.173728 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.174569 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.174572 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.175199 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.175200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.175285 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.175288 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.175355 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.175395 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":41}
I20260926 05:41:41.175510 26665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.172387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41101 (local address 127.23.213.194:47657)
0926 05:41:41.172606 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.172634 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.172683 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.172808 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.172809 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.172811 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.172876 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.172881 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.173727 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.173728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.174594 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.174598 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.175223 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.175224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.175301 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.175304 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.175373 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.175413 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":192,"mutex_wait_us":57,"thread_start_us":122,"threads_started":1}
I20260926 05:41:41.175905 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcbaa055b064e73aa79fbd606d39601" candidate_uuid: "99f9c59136cf41efb2a49eeea5ad937e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b" is_pre_election: true
I20260926 05:41:41.176059 26568 raft_consensus.cc:2454] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f9c59136cf41efb2a49eeea5ad937e in term 0.
I20260926 05:41:41.176026 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcbaa055b064e73aa79fbd606d39601" candidate_uuid: "99f9c59136cf41efb2a49eeea5ad937e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d133b12692894037a711baaf6de5e1a9" is_pre_election: true
I20260926 05:41:41.176112 26295 raft_consensus.cc:2454] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 99f9c59136cf41efb2a49eeea5ad937e in term 0.
I20260926 05:41:41.176280 26367 leader_election.cc:302] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f9c59136cf41efb2a49eeea5ad937e, d133b12692894037a711baaf6de5e1a9; no voters:
I20260926 05:41:41.176426 26650 raft_consensus.cc:2781] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:41.176467 26650 raft_consensus.cc:482] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:41.176489 26650 raft_consensus.cc:3037] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.177155 26650 raft_consensus.cc:504] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.177311 26650 leader_election.cc:288] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [CANDIDATE]: Term 1 election: Requested vote from peers d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:41.177481 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcbaa055b064e73aa79fbd606d39601" candidate_uuid: "99f9c59136cf41efb2a49eeea5ad937e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b"
I20260926 05:41:41.177481 26295 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dfcbaa055b064e73aa79fbd606d39601" candidate_uuid: "99f9c59136cf41efb2a49eeea5ad937e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d133b12692894037a711baaf6de5e1a9"
I20260926 05:41:41.177551 26295 raft_consensus.cc:3037] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.177570 26568 raft_consensus.cc:3037] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.178407 26295 raft_consensus.cc:2454] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f9c59136cf41efb2a49eeea5ad937e in term 1.
I20260926 05:41:41.178408 26568 raft_consensus.cc:2454] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 99f9c59136cf41efb2a49eeea5ad937e in term 1.
I20260926 05:41:41.178592 26369 leader_election.cc:302] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813557481cb140f2af611bd9739b8e1b, 99f9c59136cf41efb2a49eeea5ad937e; no voters:
I20260926 05:41:41.178717 26650 raft_consensus.cc:2781] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:41.178892 26650 raft_consensus.cc:686] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 LEADER]: Becoming Leader. State: Replica: 99f9c59136cf41efb2a49eeea5ad937e, State: Running, Role: LEADER
I20260926 05:41:41.179009 26650 consensus_queue.cc:237] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.179534 26666 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: TxnStatusTablet state changed. Reason: New leader 99f9c59136cf41efb2a49eeea5ad937e. Latest consensus state: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.179621 26666 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:41:41.179807 26668 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:41:41.179935 26668 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:41:41.179919 26138 catalog_manager.cc:5495] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e reported cstate change: term changed from 0 to 1, leader changed from <none> to 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194). New cstate: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:41.192893 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.190088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45973 (local address 127.0.0.1:37356)
0926 05:41:41.190177 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.190191 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.190243 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.190430 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.190431 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.190432 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:41.190478 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.190480 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.191384 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.191385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.191953 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.191955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.192421 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.192422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.192473 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.192487 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.192762 (+ 275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.192769 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.192794 (+ 25us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.192834 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 05:41:41.192922 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.190178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37356 (local address 127.23.213.194:45973)
0926 05:41:41.190248 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.190250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.190261 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:41:41.190331 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.190332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.190352 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.190417 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:41.190543 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.191314 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.192033 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.192346 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.192414 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.192574 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.192695 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.192778 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.192806 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.192827 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:41:41.194805 26295 raft_consensus.cc:1261] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Refusing update from remote peer 99f9c59136cf41efb2a49eeea5ad937e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:41.194814 26568 raft_consensus.cc:1261] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Refusing update from remote peer 99f9c59136cf41efb2a49eeea5ad937e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:41.195108 26650 consensus_queue.cc:1035] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [LEADER]: Connected to new peer: Peer: permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:41.195286 26666 consensus_queue.cc:1035] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:41.196528 26652 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: TxnStatusTablet state changed. Reason: New leader 99f9c59136cf41efb2a49eeea5ad937e. Latest consensus state: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.196619 26652 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:41.197988 26651 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: TxnStatusTablet state changed. Reason: New leader 99f9c59136cf41efb2a49eeea5ad937e. Latest consensus state: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.198077 26651 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:41.198549 26671 mvcc.cc:205] Tried to move back new op lower bound from 7333483729691009024 to 7333483729629888512. Current Snapshot: MvccSnapshot[applied={T|T < 7333483729691009024}]
I20260926 05:41:41.198668 26650 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.198740 26666 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.198830 26666 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:41:41.198747 26650 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P 99f9c59136cf41efb2a49eeea5ad937e: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:41:41.199079 26672 mvcc.cc:205] Tried to move back new op lower bound from 7333483729691009024 to 7333483729629888512. Current Snapshot: MvccSnapshot[applied={T|T < 7333483729691009024}]
I20260926 05:41:41.199270 26652 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.199349 26652 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P d133b12692894037a711baaf6de5e1a9: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:41.199832 26651 tablet_replica.cc:439] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "99f9c59136cf41efb2a49eeea5ad937e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } } }
I20260926 05:41:41.199913 26651 tablet_replica.cc:442] T dfcbaa055b064e73aa79fbd606d39601 P 813557481cb140f2af611bd9739b8e1b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:41:41.205984 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.203479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47936 (local address 127.23.213.195:37967)
0926 05:41:41.203520 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.203523 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.203531 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:41:41.203608 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.203609 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.203627 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.203701 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:41.203867 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.204348 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.205137 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.205352 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.205430 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.205605 (+ 175us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.205715 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.205849 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.205884 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.205902 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:41:41.206605 26342 tablet.cc:2369] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:41:41.211262 26627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.208468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45973 (local address 127.23.213.195:55627)
0926 05:41:41.208561 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.208571 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.208610 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.208796 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.208797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.208799 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:41.208848 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.208849 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.209795 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.209796 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.210359 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.210361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.210884 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.210885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.210947 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.210960 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.211140 (+ 180us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.211142 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.211178 (+ 36us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.211212 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 05:41:41.211297 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.208505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:55627 (local address 127.23.213.194:45973)
0926 05:41:41.208682 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.208684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.208691 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:41:41.208712 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.208713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.208722 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.208780 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:41.208915 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.209675 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.210484 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.210796 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.210866 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.211048 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.211080 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.211134 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.211210 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.211229 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260926 05:41:41.213850 26295 raft_consensus.cc:1261] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Refusing update from remote peer 813557481cb140f2af611bd9739b8e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:41.213960 26435 raft_consensus.cc:1261] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Refusing update from remote peer 813557481cb140f2af611bd9739b8e1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:41.214094 26654 consensus_queue.cc:1035] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:41.214214 26651 consensus_queue.cc:1035] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:41.215572 26674 mvcc.cc:205] Tried to move back new op lower bound from 7333483729770180608 to 7333483729313726464. Current Snapshot: MvccSnapshot[applied={T|T < 7333483729770180608}]
I20260926 05:41:41.222604 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.219848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:37967 (local address 127.23.213.194:37595)
0926 05:41:41.219920 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.219932 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.219982 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.220105 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.220105 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.220117 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:41.220172 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.220173 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.221020 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.221020 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.221840 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.221842 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.222258 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.222259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.222309 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.222322 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.222488 (+ 166us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.222489 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.222520 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.222551 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:41:41.222612 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.219903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:37595 (local address 127.23.213.195:37967)
0926 05:41:41.219940 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.219942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.219966 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:41:41.220031 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.220032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.220041 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.220102 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:41.220222 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.220943 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.221907 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.222184 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.222253 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.222400 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.222433 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.222490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.222523 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.222543 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:41:41.231936 26138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:41.232064 26138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:41.232842 26138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Sending DeleteTablet for 3 replicas of tablet b7e0d9ff0d24434e884e1741d83e7e4d
I20260926 05:41:41.233265 26275 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e0d9ff0d24434e884e1741d83e7e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:41 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43276
I20260926 05:41:41.233385 26138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52412:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:41:41.233502 26415 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e0d9ff0d24434e884e1741d83e7e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:41 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37354
W20260926 05:41:41.233536 26138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:41.233793 26555 tablet_service.cc:1514] Processing DeleteTablet for tablet b7e0d9ff0d24434e884e1741d83e7e4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:41 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:47920
I20260926 05:41:41.234006 26689 tablet_replica.cc:330] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: stopping tablet replica
I20260926 05:41:41.234122 26689 raft_consensus.cc:2229] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:41.234192 26689 raft_consensus.cc:2258] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:41.234359 26690 tablet_replica.cc:330] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: stopping tablet replica
I20260926 05:41:41.234458 26690 raft_consensus.cc:2229] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:41.234513 26690 raft_consensus.cc:2258] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:41.234966 26689 ts_tablet_manager.cc:1905] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:41.235287 26275 tablet_service.cc:1467] Processing CreateTablet for tablet 9e269ab713ab422d88e6345fb3a70d3e (DEFAULT_TABLE table=test-table [id=859175690eb9434bb6a6859c7142228c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:41.235376 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e269ab713ab422d88e6345fb3a70d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.236455 26690 ts_tablet_manager.cc:1905] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:41.236617 26689 ts_tablet_manager.cc:1918] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:41.236680 26689 log.cc:1196] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/b7e0d9ff0d24434e884e1741d83e7e4d
I20260926 05:41:41.236649 26415 tablet_service.cc:1467] Processing CreateTablet for tablet 9e269ab713ab422d88e6345fb3a70d3e (DEFAULT_TABLE table=test-table [id=859175690eb9434bb6a6859c7142228c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:41.236761 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e269ab713ab422d88e6345fb3a70d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.236937 26689 ts_tablet_manager.cc:1939] T b7e0d9ff0d24434e884e1741d83e7e4d P d133b12692894037a711baaf6de5e1a9: Deleting consensus metadata
I20260926 05:41:41.238013 26644 tablet_bootstrap.cc:492] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: Bootstrap starting.
I20260926 05:41:41.238078 26555 tablet_service.cc:1467] Processing CreateTablet for tablet 9e269ab713ab422d88e6345fb3a70d3e (DEFAULT_TABLE table=test-table [id=859175690eb9434bb6a6859c7142228c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:41.238215 26555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9e269ab713ab422d88e6345fb3a70d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:41.238220 26690 ts_tablet_manager.cc:1918] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:41.238283 26690 log.cc:1196] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/b7e0d9ff0d24434e884e1741d83e7e4d
I20260926 05:41:41.238418 26644 tablet_bootstrap.cc:654] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.238564 26690 ts_tablet_manager.cc:1939] T b7e0d9ff0d24434e884e1741d83e7e4d P 99f9c59136cf41efb2a49eeea5ad937e: Deleting consensus metadata
I20260926 05:41:41.238981 26122 catalog_manager.cc:4841] TS 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973): tablet b7e0d9ff0d24434e884e1741d83e7e4d (table test-table [id=4368fc62e42e4cac9dae891b78fff4db]) successfully deleted
I20260926 05:41:41.239084 26644 tablet_bootstrap.cc:492] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: No bootstrap required, opened a new log
I20260926 05:41:41.239130 26644 ts_tablet_manager.cc:1397] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:41.239280 26644 raft_consensus.cc:348] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.239341 26644 raft_consensus.cc:374] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.239359 26644 raft_consensus.cc:729] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f9c59136cf41efb2a49eeea5ad937e, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.239392 26644 consensus_queue.cc:260] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.239481 26644 ts_tablet_manager.cc:1428] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:41.239681 26645 tablet_bootstrap.cc:492] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Bootstrap starting.
I20260926 05:41:41.239719 26646 tablet_bootstrap.cc:492] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: Bootstrap starting.
I20260926 05:41:41.240096 26645 tablet_bootstrap.cc:654] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.240121 26646 tablet_bootstrap.cc:654] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:41.240649 26123 catalog_manager.cc:4841] TS d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101): tablet b7e0d9ff0d24434e884e1741d83e7e4d (table test-table [id=4368fc62e42e4cac9dae891b78fff4db]) successfully deleted
I20260926 05:41:41.241335 26646 tablet_bootstrap.cc:492] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: No bootstrap required, opened a new log
I20260926 05:41:41.241385 26646 ts_tablet_manager.cc:1397] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:41.241532 26646 raft_consensus.cc:348] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.241585 26646 raft_consensus.cc:374] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.241598 26646 raft_consensus.cc:729] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.241657 26646 consensus_queue.cc:260] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.241791 26646 ts_tablet_manager.cc:1428] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:41.245854 26691 tablet_replica.cc:330] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: stopping tablet replica
I20260926 05:41:41.245994 26691 raft_consensus.cc:2229] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:41:41.246102 26691 raft_consensus.cc:2258] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:41.246219 26645 tablet_bootstrap.cc:492] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: No bootstrap required, opened a new log
I20260926 05:41:41.246271 26645 ts_tablet_manager.cc:1397] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.000s
I20260926 05:41:41.246424 26645 raft_consensus.cc:348] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.246515 26645 raft_consensus.cc:374] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:41.246543 26645 raft_consensus.cc:729] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813557481cb140f2af611bd9739b8e1b, State: Initialized, Role: FOLLOWER
I20260926 05:41:41.246650 26645 consensus_queue.cc:260] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.246776 26645 ts_tablet_manager.cc:1428] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:41.246541 26691 ts_tablet_manager.cc:1905] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:41.248417 26691 ts_tablet_manager.cc:1918] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:41.248500 26691 log.cc:1196] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/b7e0d9ff0d24434e884e1741d83e7e4d
I20260926 05:41:41.248759 26691 ts_tablet_manager.cc:1939] T b7e0d9ff0d24434e884e1741d83e7e4d P 813557481cb140f2af611bd9739b8e1b: Deleting consensus metadata
I20260926 05:41:41.249146 26125 catalog_manager.cc:4841] TS 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967): tablet b7e0d9ff0d24434e884e1741d83e7e4d (table test-table [id=4368fc62e42e4cac9dae891b78fff4db]) successfully deleted
W20260926 05:41:41.275604 26622 tablet.cc:2369] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:41:41.411301 26652 raft_consensus.cc:482] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:41.411396 26652 raft_consensus.cc:504] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.411782 26652 leader_election.cc:288] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:41.414861 26642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.412048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:37539 (local address 127.23.213.195:37967)
0926 05:41:41.412098 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.412100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.412236 (+ 136us) server_negotiation.cc:407] Connection header received
0926 05:41:41.412309 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.412310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.412324 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.412396 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.412567 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.413049 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.414151 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.414504 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.414598 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.414607 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.414751 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.414786 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:41:41.414880 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.412119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:58711 (local address 127.23.213.194:45973)
0926 05:41:41.412182 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.412184 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.412210 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:41:41.412296 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.412297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.412307 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.412389 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:41.412599 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.413054 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.414152 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.414514 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.414607 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.414615 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.414776 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.414810 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:41:41.414907 26695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.411986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:37967 (local address 127.23.213.193:37539)
0926 05:41:41.412130 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.412160 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.412250 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.412400 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.412401 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.412402 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.412489 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.412491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.413131 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.413132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.414058 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.414060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.414599 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.414600 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.414675 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.414679 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.414745 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.414784 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":46,"thread_start_us":62,"threads_started":1}
I20260926 05:41:41.414917 26696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.411987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45973 (local address 127.23.213.193:58711)
0926 05:41:41.412151 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.412162 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.412230 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.412396 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.412397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.412399 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:41.412515 (+ 116us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.412516 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.413141 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.413142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.414062 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.414065 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.414608 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.414609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.414690 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.414693 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.414769 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.414809 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":47,"thread_start_us":58,"threads_started":1}
I20260926 05:41:41.415288 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e269ab713ab422d88e6345fb3a70d3e" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b" is_pre_election: true
I20260926 05:41:41.415390 26568 raft_consensus.cc:2454] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 0.
I20260926 05:41:41.415397 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e269ab713ab422d88e6345fb3a70d3e" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e" is_pre_election: true
I20260926 05:41:41.415505 26435 raft_consensus.cc:2454] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 0.
I20260926 05:41:41.415541 26229 leader_election.cc:302] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813557481cb140f2af611bd9739b8e1b, d133b12692894037a711baaf6de5e1a9; no voters:
I20260926 05:41:41.415663 26652 raft_consensus.cc:2781] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:41.415705 26652 raft_consensus.cc:482] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:41.415720 26652 raft_consensus.cc:3037] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.416396 26652 raft_consensus.cc:504] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.416531 26652 leader_election.cc:288] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 election: Requested vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:41.416682 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e269ab713ab422d88e6345fb3a70d3e" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e"
I20260926 05:41:41.416760 26435 raft_consensus.cc:3037] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.417361 26435 raft_consensus.cc:2454] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 1.
I20260926 05:41:41.417512 26568 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9e269ab713ab422d88e6345fb3a70d3e" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b"
I20260926 05:41:41.417578 26226 leader_election.cc:302] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f9c59136cf41efb2a49eeea5ad937e, d133b12692894037a711baaf6de5e1a9; no voters:
I20260926 05:41:41.417609 26568 raft_consensus.cc:3037] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:41.417667 26652 raft_consensus.cc:2781] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:41.417838 26652 raft_consensus.cc:686] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 1 LEADER]: Becoming Leader. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Running, Role: LEADER
I20260926 05:41:41.417901 26652 consensus_queue.cc:237] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:41.418517 26568 raft_consensus.cc:2454] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 1.
I20260926 05:41:41.418651 26138 catalog_manager.cc:5495] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d133b12692894037a711baaf6de5e1a9 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "d133b12692894037a711baaf6de5e1a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:41.427625 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.425497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43292 (local address 127.23.213.193:41101)
0926 05:41:41.425537 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.425539 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.425582 (+ 43us) server_negotiation.cc:407] Connection header received
0926 05:41:41.425633 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.425634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.425650 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.425720 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:41.425857 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.426297 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.426958 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.427129 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.427179 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.427305 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.427403 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.427456 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.427491 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.427505 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:41:41.431960 26641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.429936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:40599 (local address 127.23.213.194:45973)
0926 05:41:41.429980 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.429982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.430078 (+ 96us) server_negotiation.cc:407] Connection header received
0926 05:41:41.430102 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.430103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.430114 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.430159 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:41.430277 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.430712 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.431385 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.431555 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.431601 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.431750 (+ 149us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.431783 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.431839 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.431894 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.431911 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:41:41.432143 26701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.429853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45973 (local address 127.23.213.193:40599)
0926 05:41:41.430012 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.430021 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.430070 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.430172 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.430173 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.430174 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:41.430225 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.430226 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.430770 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.430771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.431321 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.431322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.431616 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.431616 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.431666 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.431678 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.431848 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.431850 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.431880 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.431909 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260926 05:41:41.434250 26568 raft_consensus.cc:1261] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Refusing update from remote peer d133b12692894037a711baaf6de5e1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:41.434250 26435 raft_consensus.cc:1261] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Refusing update from remote peer d133b12692894037a711baaf6de5e1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:41.434527 26697 consensus_queue.cc:1035] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:41.434686 26652 consensus_queue.cc:1035] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:41.441498 26643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.439176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:60381 (local address 127.23.213.193:41101)
0926 05:41:41.439287 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:41.439289 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:41.439296 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:41:41.439320 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:41.439320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:41.439330 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:41.439384 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:41.439520 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.440147 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.440827 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.441010 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.441094 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.441264 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.441298 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.441342 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:41.441411 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 05:41:41.441431 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 05:41:41.441529 26487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:41.439093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41101 (local address 127.23.213.194:60381)
0926 05:41:41.439134 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:41.439145 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:41.439191 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:41.439412 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:41.439413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:41.439414 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:41:41.439460 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.439462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.440218 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.440219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.440752 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:41.440754 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:41.441102 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:41.441103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:41.441177 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:41.441182 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:41:41.441358 (+ 176us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:41:41.441361 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:41.441410 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:41:41.441457 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 05:41:41.441828 26672 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 05:41:42.961781 26138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:42.961907 26138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52412:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:42.962698 26138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Sending DeleteTablet for 3 replicas of tablet 9e269ab713ab422d88e6345fb3a70d3e
I20260926 05:41:42.962945 26415 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e269ab713ab422d88e6345fb3a70d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:42 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260926 05:41:42.962961 26275 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e269ab713ab422d88e6345fb3a70d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:42 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43276
I20260926 05:41:42.963209 26555 tablet_service.cc:1514] Processing DeleteTablet for tablet 9e269ab713ab422d88e6345fb3a70d3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:42 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:47920
I20260926 05:41:42.963168 26728 tablet_replica.cc:330] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: stopping tablet replica
I20260926 05:41:42.963426 26730 tablet_replica.cc:330] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: stopping tablet replica
I20260926 05:41:42.963460 26728 raft_consensus.cc:2229] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:41:42.963472 26730 raft_consensus.cc:2229] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:42.963505 26730 raft_consensus.cc:2258] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:42.963577 26728 raft_consensus.cc:2258] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:42.963812 26730 ts_tablet_manager.cc:1905] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:42.963812 26728 ts_tablet_manager.cc:1905] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:42.965545 26729 tablet_replica.cc:330] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: stopping tablet replica
I20260926 05:41:42.965545 26730 ts_tablet_manager.cc:1918] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:41:42.965603 26730 log.cc:1196] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/9e269ab713ab422d88e6345fb3a70d3e
I20260926 05:41:42.965720 26728 ts_tablet_manager.cc:1918] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:41:42.965814 26728 log.cc:1196] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/9e269ab713ab422d88e6345fb3a70d3e
I20260926 05:41:42.965857 26730 ts_tablet_manager.cc:1939] T 9e269ab713ab422d88e6345fb3a70d3e P 813557481cb140f2af611bd9739b8e1b: Deleting consensus metadata
I20260926 05:41:42.966060 26728 ts_tablet_manager.cc:1939] T 9e269ab713ab422d88e6345fb3a70d3e P d133b12692894037a711baaf6de5e1a9: Deleting consensus metadata
I20260926 05:41:42.966351 26125 catalog_manager.cc:4841] TS 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967): tablet 9e269ab713ab422d88e6345fb3a70d3e (table test-table [id=859175690eb9434bb6a6859c7142228c]) successfully deleted
I20260926 05:41:42.966384 26123 catalog_manager.cc:4841] TS d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101): tablet 9e269ab713ab422d88e6345fb3a70d3e (table test-table [id=859175690eb9434bb6a6859c7142228c]) successfully deleted
I20260926 05:41:42.968400 26729 raft_consensus.cc:2229] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:42.968482 26729 raft_consensus.cc:2258] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:42.968716 26729 ts_tablet_manager.cc:1905] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:42.970311 26729 ts_tablet_manager.cc:1918] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:41:42.970379 26729 log.cc:1196] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/9e269ab713ab422d88e6345fb3a70d3e
I20260926 05:41:42.970556 26729 ts_tablet_manager.cc:1939] T 9e269ab713ab422d88e6345fb3a70d3e P 99f9c59136cf41efb2a49eeea5ad937e: Deleting consensus metadata
I20260926 05:41:42.971000 26122 catalog_manager.cc:4841] TS 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973): tablet 9e269ab713ab422d88e6345fb3a70d3e (table test-table [id=859175690eb9434bb6a6859c7142228c]) successfully deleted
Sep 26 05:41:42 dist-test-slave-1fdt krb5kdc[26093](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:41:42 dist-test-slave-1fdt krb5kdc[26093](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401302, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:42.987193 26738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:42.979949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52422 (local address 127.23.213.254:41699)
0926 05:41:42.980090 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:42.980093 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:42.980154 (+ 61us) server_negotiation.cc:407] Connection header received
0926 05:41:42.980252 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:42.980253 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:42.980265 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:42.980362 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:42.980959 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:42.981498 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:42.982317 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:42.982535 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:42.982621 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:42.984799 (+ 2178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:42.984800 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:42.984800 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:42.984807 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:42.986116 (+ 1309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:42.986437 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:42.986437 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:42.986438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:42.986509 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:42.986763 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:42.986763 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:42.986764 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:42.986866 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:42.986942 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:42.987048 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 05:41:42.987112 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":73,"threads_started":1}
I20260926 05:41:42.988623 26138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:41:42.988770 26138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:42.990041 26275 tablet_service.cc:1467] Processing CreateTablet for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 (DEFAULT_TABLE table=test-table [id=dcadb40963a945069fff6e44687679db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:42.990085 26415 tablet_service.cc:1467] Processing CreateTablet for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 (DEFAULT_TABLE table=test-table [id=dcadb40963a945069fff6e44687679db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:42.990180 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:42.990142 26555 tablet_service.cc:1467] Processing CreateTablet for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 (DEFAULT_TABLE table=test-table [id=dcadb40963a945069fff6e44687679db]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:42.990216 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:42.990263 26555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:42.991187 26739 tablet_bootstrap.cc:492] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: Bootstrap starting.
I20260926 05:41:42.991595 26739 tablet_bootstrap.cc:654] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:42.991670 26741 tablet_bootstrap.cc:492] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: Bootstrap starting.
I20260926 05:41:42.991994 26741 tablet_bootstrap.cc:654] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:42.992208 26739 tablet_bootstrap.cc:492] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: No bootstrap required, opened a new log
I20260926 05:41:42.992249 26739 ts_tablet_manager.cc:1397] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:42.992332 26740 tablet_bootstrap.cc:492] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: Bootstrap starting.
I20260926 05:41:42.992420 26739 raft_consensus.cc:348] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:42.992523 26739 raft_consensus.cc:374] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:42.992573 26739 raft_consensus.cc:729] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Initialized, Role: FOLLOWER
I20260926 05:41:42.992646 26739 consensus_queue.cc:260] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:42.992651 26741 tablet_bootstrap.cc:492] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: No bootstrap required, opened a new log
I20260926 05:41:42.992717 26740 tablet_bootstrap.cc:654] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:42.992769 26739 ts_tablet_manager.cc:1428] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:42.992713 26741 ts_tablet_manager.cc:1397] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:42.993012 26741 raft_consensus.cc:348] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:42.993088 26741 raft_consensus.cc:374] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:42.993112 26741 raft_consensus.cc:729] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813557481cb140f2af611bd9739b8e1b, State: Initialized, Role: FOLLOWER
I20260926 05:41:42.993197 26741 consensus_queue.cc:260] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:42.993376 26741 ts_tablet_manager.cc:1428] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:42.993412 26740 tablet_bootstrap.cc:492] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: No bootstrap required, opened a new log
I20260926 05:41:42.993501 26740 ts_tablet_manager.cc:1397] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:42.993633 26740 raft_consensus.cc:348] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:42.993736 26740 raft_consensus.cc:374] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:42.993778 26740 raft_consensus.cc:729] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f9c59136cf41efb2a49eeea5ad937e, State: Initialized, Role: FOLLOWER
I20260926 05:41:42.993832 26740 consensus_queue.cc:260] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:42.993965 26740 ts_tablet_manager.cc:1428] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:43.148548 26720 raft_consensus.cc:482] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:43.148686 26720 raft_consensus.cc:504] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.148921 26720 leader_election.cc:288] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:43.149176 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b86ff2ec81b4ab39a0ae213a458f6a6" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e" is_pre_election: true
I20260926 05:41:43.149190 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b86ff2ec81b4ab39a0ae213a458f6a6" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b" is_pre_election: true
I20260926 05:41:43.149276 26435 raft_consensus.cc:2454] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 0.
I20260926 05:41:43.149300 26575 raft_consensus.cc:2454] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 0.
I20260926 05:41:43.149479 26229 leader_election.cc:302] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813557481cb140f2af611bd9739b8e1b, d133b12692894037a711baaf6de5e1a9; no voters:
I20260926 05:41:43.149592 26720 raft_consensus.cc:2781] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:43.149641 26720 raft_consensus.cc:482] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:43.149655 26720 raft_consensus.cc:3037] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:43.150381 26720 raft_consensus.cc:504] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.150522 26720 leader_election.cc:288] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 election: Requested vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:43.150698 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b86ff2ec81b4ab39a0ae213a458f6a6" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e"
I20260926 05:41:43.150763 26435 raft_consensus.cc:3037] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:43.150697 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5b86ff2ec81b4ab39a0ae213a458f6a6" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b"
I20260926 05:41:43.150792 26575 raft_consensus.cc:3037] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:43.151322 26435 raft_consensus.cc:2454] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 1.
I20260926 05:41:43.151526 26575 raft_consensus.cc:2454] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 1.
I20260926 05:41:43.151515 26226 leader_election.cc:302] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f9c59136cf41efb2a49eeea5ad937e, d133b12692894037a711baaf6de5e1a9; no voters:
I20260926 05:41:43.151679 26720 raft_consensus.cc:2781] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:43.151746 26720 raft_consensus.cc:686] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 1 LEADER]: Becoming Leader. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Running, Role: LEADER
I20260926 05:41:43.151804 26720 consensus_queue.cc:237] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.152518 26138 catalog_manager.cc:5495] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d133b12692894037a711baaf6de5e1a9 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "d133b12692894037a711baaf6de5e1a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:43.177348 26748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:43.175224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43308 (local address 127.23.213.193:41101)
0926 05:41:43.175379 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:43.175381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:43.175389 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:41:43.175417 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:43.175418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:43.175428 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:43.175486 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:43.175597 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:43.176082 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:43.176798 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:43.177000 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:43.177050 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:43.177160 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:43.177185 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:43.177218 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:43.177269 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 05:41:43.177286 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260926 05:41:43.179848 26575 raft_consensus.cc:1261] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Refusing update from remote peer d133b12692894037a711baaf6de5e1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:43.179915 26435 raft_consensus.cc:1261] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Refusing update from remote peer d133b12692894037a711baaf6de5e1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:43.180075 26707 consensus_queue.cc:1035] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:43.180174 26720 consensus_queue.cc:1035] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:43.189940 26138 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:43.190032 26138 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:43.190614 26138 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Sending DeleteTablet for 3 replicas of tablet 5b86ff2ec81b4ab39a0ae213a458f6a6
I20260926 05:41:43.190868 26415 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:43 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260926 05:41:43.190872 26275 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:43 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43276
I20260926 05:41:43.190909 26555 tablet_service.cc:1514] Processing DeleteTablet for tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:43 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:47920
I20260926 05:41:43.190964 26729 tablet_replica.cc:330] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: stopping tablet replica
I20260926 05:41:43.190994 26730 tablet_replica.cc:330] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: stopping tablet replica
I20260926 05:41:43.191008 26728 tablet_replica.cc:330] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: stopping tablet replica
I20260926 05:41:43.191020 26729 raft_consensus.cc:2229] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:43.191044 26730 raft_consensus.cc:2229] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:43.191059 26729 raft_consensus.cc:2258] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:43.191082 26730 raft_consensus.cc:2258] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:43.191059 26728 raft_consensus.cc:2229] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:41:43.191133 26728 raft_consensus.cc:2258] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:43.191442 26730 ts_tablet_manager.cc:1905] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:43.191442 26728 ts_tablet_manager.cc:1905] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:43.191452 26729 ts_tablet_manager.cc:1905] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:43.192785 26729 ts_tablet_manager.cc:1918] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:43.192852 26729 log.cc:1196] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/5b86ff2ec81b4ab39a0ae213a458f6a6
I20260926 05:41:43.193061 26728 ts_tablet_manager.cc:1918] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:43.193066 26730 ts_tablet_manager.cc:1918] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:41:43.193078 26729 ts_tablet_manager.cc:1939] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 99f9c59136cf41efb2a49eeea5ad937e: Deleting consensus metadata
I20260926 05:41:43.193109 26728 log.cc:1196] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/5b86ff2ec81b4ab39a0ae213a458f6a6
I20260926 05:41:43.193115 26730 log.cc:1196] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/5b86ff2ec81b4ab39a0ae213a458f6a6
I20260926 05:41:43.193049 26138 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:41:43.193202 26138 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:43.193318 26728 ts_tablet_manager.cc:1939] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P d133b12692894037a711baaf6de5e1a9: Deleting consensus metadata
I20260926 05:41:43.193354 26730 ts_tablet_manager.cc:1939] T 5b86ff2ec81b4ab39a0ae213a458f6a6 P 813557481cb140f2af611bd9739b8e1b: Deleting consensus metadata
I20260926 05:41:43.193640 26123 catalog_manager.cc:4841] TS d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101): tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 (table test-table [id=dcadb40963a945069fff6e44687679db]) successfully deleted
I20260926 05:41:43.193691 26125 catalog_manager.cc:4841] TS 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967): tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 (table test-table [id=dcadb40963a945069fff6e44687679db]) successfully deleted
I20260926 05:41:43.193785 26122 catalog_manager.cc:4841] TS 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973): tablet 5b86ff2ec81b4ab39a0ae213a458f6a6 (table test-table [id=dcadb40963a945069fff6e44687679db]) successfully deleted
I20260926 05:41:43.194411 26275 tablet_service.cc:1467] Processing CreateTablet for tablet 55de740e54634356ae65989ae09fdf30 (DEFAULT_TABLE table=test-table [id=9ffbfeb0de5f4c8397a122d6e75bd951]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:43.194445 26415 tablet_service.cc:1467] Processing CreateTablet for tablet 55de740e54634356ae65989ae09fdf30 (DEFAULT_TABLE table=test-table [id=9ffbfeb0de5f4c8397a122d6e75bd951]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:43.194456 26555 tablet_service.cc:1467] Processing CreateTablet for tablet 55de740e54634356ae65989ae09fdf30 (DEFAULT_TABLE table=test-table [id=9ffbfeb0de5f4c8397a122d6e75bd951]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:43.194511 26275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55de740e54634356ae65989ae09fdf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:43.194523 26415 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55de740e54634356ae65989ae09fdf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:43.194548 26555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 55de740e54634356ae65989ae09fdf30. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:43.195313 26741 tablet_bootstrap.cc:492] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: Bootstrap starting.
I20260926 05:41:43.195626 26739 tablet_bootstrap.cc:492] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9: Bootstrap starting.
I20260926 05:41:43.195674 26741 tablet_bootstrap.cc:654] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:43.195936 26739 tablet_bootstrap.cc:654] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:43.195976 26740 tablet_bootstrap.cc:492] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: Bootstrap starting.
I20260926 05:41:43.196312 26740 tablet_bootstrap.cc:654] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:43.196434 26739 tablet_bootstrap.cc:492] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9: No bootstrap required, opened a new log
I20260926 05:41:43.196488 26739 ts_tablet_manager.cc:1397] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:43.196651 26741 tablet_bootstrap.cc:492] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: No bootstrap required, opened a new log
I20260926 05:41:43.196605 26739 raft_consensus.cc:348] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.196689 26739 raft_consensus.cc:374] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:43.196702 26741 ts_tablet_manager.cc:1397] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:43.196715 26739 raft_consensus.cc:729] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Initialized, Role: FOLLOWER
I20260926 05:41:43.196765 26739 consensus_queue.cc:260] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.196800 26741 raft_consensus.cc:348] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.196863 26741 raft_consensus.cc:374] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:43.196863 26739 ts_tablet_manager.cc:1428] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:43.196892 26741 raft_consensus.cc:729] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 813557481cb140f2af611bd9739b8e1b, State: Initialized, Role: FOLLOWER
I20260926 05:41:43.196938 26741 consensus_queue.cc:260] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.197033 26741 ts_tablet_manager.cc:1428] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:43.197504 26740 tablet_bootstrap.cc:492] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: No bootstrap required, opened a new log
I20260926 05:41:43.197563 26740 ts_tablet_manager.cc:1397] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:43.197710 26740 raft_consensus.cc:348] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.197818 26740 raft_consensus.cc:374] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:43.197852 26740 raft_consensus.cc:729] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 99f9c59136cf41efb2a49eeea5ad937e, State: Initialized, Role: FOLLOWER
I20260926 05:41:43.197906 26740 consensus_queue.cc:260] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.198025 26740 ts_tablet_manager.cc:1428] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:43.330363 26752 raft_consensus.cc:482] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:43.330484 26752 raft_consensus.cc:504] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.330706 26752 leader_election.cc:288] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:43.330962 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de740e54634356ae65989ae09fdf30" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b" is_pre_election: true
I20260926 05:41:43.331046 26575 raft_consensus.cc:2454] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 0.
I20260926 05:41:43.330964 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de740e54634356ae65989ae09fdf30" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e" is_pre_election: true
I20260926 05:41:43.331086 26435 raft_consensus.cc:2454] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 0.
I20260926 05:41:43.331221 26229 leader_election.cc:302] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 813557481cb140f2af611bd9739b8e1b, d133b12692894037a711baaf6de5e1a9; no voters:
I20260926 05:41:43.331341 26752 raft_consensus.cc:2781] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:43.331396 26752 raft_consensus.cc:482] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:43.331411 26752 raft_consensus.cc:3037] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:43.332098 26752 raft_consensus.cc:504] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.332245 26752 leader_election.cc:288] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 election: Requested vote from peers 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973), 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967)
I20260926 05:41:43.332396 26575 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de740e54634356ae65989ae09fdf30" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "813557481cb140f2af611bd9739b8e1b"
I20260926 05:41:43.332396 26435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "55de740e54634356ae65989ae09fdf30" candidate_uuid: "d133b12692894037a711baaf6de5e1a9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "99f9c59136cf41efb2a49eeea5ad937e"
I20260926 05:41:43.332466 26575 raft_consensus.cc:3037] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:43.332486 26435 raft_consensus.cc:3037] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:43.333087 26435 raft_consensus.cc:2454] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 1.
I20260926 05:41:43.333140 26575 raft_consensus.cc:2454] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d133b12692894037a711baaf6de5e1a9 in term 1.
I20260926 05:41:43.333289 26226 leader_election.cc:302] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 99f9c59136cf41efb2a49eeea5ad937e, d133b12692894037a711baaf6de5e1a9; no voters:
I20260926 05:41:43.333391 26752 raft_consensus.cc:2781] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:43.333458 26752 raft_consensus.cc:686] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [term 1 LEADER]: Becoming Leader. State: Replica: d133b12692894037a711baaf6de5e1a9, State: Running, Role: LEADER
I20260926 05:41:43.333498 26752 consensus_queue.cc:237] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } }
I20260926 05:41:43.334070 26138 catalog_manager.cc:5495] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 reported cstate change: term changed from 0 to 1, leader changed from <none> to d133b12692894037a711baaf6de5e1a9 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "d133b12692894037a711baaf6de5e1a9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d133b12692894037a711baaf6de5e1a9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:43.344353 26435 raft_consensus.cc:1261] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Refusing update from remote peer d133b12692894037a711baaf6de5e1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:43.344380 26575 raft_consensus.cc:1261] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Refusing update from remote peer d133b12692894037a711baaf6de5e1a9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:43.344622 26752 consensus_queue.cc:1035] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "99f9c59136cf41efb2a49eeea5ad937e" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45973 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:43.344790 26707 consensus_queue.cc:1035] T 55de740e54634356ae65989ae09fdf30 P d133b12692894037a711baaf6de5e1a9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "813557481cb140f2af611bd9739b8e1b" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37967 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:41:43.349171 26711 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 05:41:44.866359 26136 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:44.866474 26136 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:44.867239 26136 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5632a45822347f084b440ca1cbf89b0: Sending DeleteTablet for 3 replicas of tablet 55de740e54634356ae65989ae09fdf30
I20260926 05:41:44.867534 26414 tablet_service.cc:1514] Processing DeleteTablet for tablet 55de740e54634356ae65989ae09fdf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:44 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37354
I20260926 05:41:44.867545 26275 tablet_service.cc:1514] Processing DeleteTablet for tablet 55de740e54634356ae65989ae09fdf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:44 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43276
I20260926 05:41:44.867736 26782 tablet_replica.cc:330] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: stopping tablet replica
I20260926 05:41:44.867753 26555 tablet_service.cc:1514] Processing DeleteTablet for tablet 55de740e54634356ae65989ae09fdf30 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:44 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:47920
I20260926 05:41:44.867837 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 26210
I20260926 05:41:44.867923 26784 tablet_replica.cc:330] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: stopping tablet replica
I20260926 05:41:44.868445 26784 raft_consensus.cc:2229] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:44.868484 26784 raft_consensus.cc:2258] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:44.868594 26782 raft_consensus.cc:2229] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:44.868649 26782 raft_consensus.cc:2258] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:44.868743 26784 ts_tablet_manager.cc:1905] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:44.868886 26782 ts_tablet_manager.cc:1905] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:44.870543 26784 ts_tablet_manager.cc:1918] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:41:44.870633 26784 log.cc:1196] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/55de740e54634356ae65989ae09fdf30
I20260926 05:41:44.870826 26782 ts_tablet_manager.cc:1918] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 05:41:44.870894 26782 log.cc:1196] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/55de740e54634356ae65989ae09fdf30
I20260926 05:41:44.870893 26784 ts_tablet_manager.cc:1939] T 55de740e54634356ae65989ae09fdf30 P 813557481cb140f2af611bd9739b8e1b: Deleting consensus metadata
I20260926 05:41:44.871169 26782 ts_tablet_manager.cc:1939] T 55de740e54634356ae65989ae09fdf30 P 99f9c59136cf41efb2a49eeea5ad937e: Deleting consensus metadata
I20260926 05:41:44.871212 26125 catalog_manager.cc:4841] TS 813557481cb140f2af611bd9739b8e1b (127.23.213.195:37967): tablet 55de740e54634356ae65989ae09fdf30 (table test-table [id=9ffbfeb0de5f4c8397a122d6e75bd951]) successfully deleted
I20260926 05:41:44.871485 26122 catalog_manager.cc:4841] TS 99f9c59136cf41efb2a49eeea5ad937e (127.23.213.194:45973): tablet 55de740e54634356ae65989ae09fdf30 (table test-table [id=9ffbfeb0de5f4c8397a122d6e75bd951]) successfully deleted
W20260926 05:41:44.874773 26123 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:41101 (error 108)
I20260926 05:41:44.875075 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 26350
I20260926 05:41:44.877930 26786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:44.875154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41101 (local address 127.0.0.1:43324)
0926 05:41:44.877835 (+ 2681us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:44.877898 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:41101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2654,"thread_start_us":60,"threads_started":1}
W20260926 05:41:44.878073 26123 catalog_manager.cc:4584] TS d133b12692894037a711baaf6de5e1a9 (127.23.213.193:41101): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 55de740e54634356ae65989ae09fdf30: Network error: Client connection negotiation failed: client connection to 127.23.213.193:41101: connect: Connection refused (error 111)
I20260926 05:41:44.881867 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 26490
I20260926 05:41:44.887485 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 26106
2026-09-26T05:41:44Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5643 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:44 dist-test-slave-1fdt krb5kdc[26789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:44 dist-test-slave-1fdt krb5kdc[26789](info): set up 2 sockets
Sep 26 05:41:44 dist-test-slave-1fdt krb5kdc[26789](info): commencing operation
krb5kdc: starting...
W20260926 05:41:45.921602 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:45 dist-test-slave-1fdt krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:46.034817 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40865
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:40865
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:46.115897 26802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:46.116107 26802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:46.116128 26802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:46.117213 26802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:46.117246 26802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:46.117260 26802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:46.117273 26802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:46.117285 26802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:46.118723 26802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:40865
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40865
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:46.118978 26802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:46.119170 26802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:46.124662 26807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:46.124691 26810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:46.124770 26802 server_base.cc:1034] running on GCE node
W20260926 05:41:46.124662 26808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:46.125247 26802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:46.125560 26802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:46.126703 26802 hybrid_clock.cc:648] HybridClock initialized: now 1790401306126685 us; error 39 us; skew 500 ppm
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:46.129331 26802 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:46.129884 26802 webserver.cc:460] Webserver started at http://127.23.213.254:35215/ using document root <none> and password file <none>
I20260926 05:41:46.130125 26802 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:46.130198 26802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:46.130349 26802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:46.131413 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "0fec35079514420384128ceb783bbf34"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "6108d2903a6267eefa8d7ee3af14cc0f"
server_key_iv: "011a13aa7be8c168eb444fb124eb45a7"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.131778 26802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "0fec35079514420384128ceb783bbf34"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "6108d2903a6267eefa8d7ee3af14cc0f"
server_key_iv: "011a13aa7be8c168eb444fb124eb45a7"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.133052 26802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:46.133831 26817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.134006 26802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:46.134104 26802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "0fec35079514420384128ceb783bbf34"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "6108d2903a6267eefa8d7ee3af14cc0f"
server_key_iv: "011a13aa7be8c168eb444fb124eb45a7"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.134194 26802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:46.151832 26802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:46.154263 26802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:46.154423 26802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:46.158540 26802 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:40865
I20260926 05:41:46.158591 26869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:40865 every 8 connection(s)
I20260926 05:41:46.159034 26802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:46.159698 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 26802
I20260926 05:41:46.159785 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:46.159984 24407 external_mini_cluster.cc:1428] Setting key 4b22f8ba10484dc4d0a754c9853ee625
I20260926 05:41:46.160156 26870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:46.163726 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34: Bootstrap starting.
I20260926 05:41:46.164429 26870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:46.164746 26870 log.cc:824] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:46.165544 26870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34: No bootstrap required, opened a new log
I20260926 05:41:46.168130 26870 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec35079514420384128ceb783bbf34" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40865 } }
I20260926 05:41:46.168282 26870 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:46.168325 26870 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0fec35079514420384128ceb783bbf34, State: Initialized, Role: FOLLOWER
I20260926 05:41:46.168452 26870 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec35079514420384128ceb783bbf34" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40865 } }
I20260926 05:41:46.168526 26870 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:46.168569 26870 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:46.168622 26870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401305, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:46.169118 26870 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec35079514420384128ceb783bbf34" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40865 } }
I20260926 05:41:46.169242 26870 leader_election.cc:302] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0fec35079514420384128ceb783bbf34; no voters:
I20260926 05:41:46.169414 26870 leader_election.cc:288] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:46.169442 26875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:46.169550 26875 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [term 1 LEADER]: Becoming Leader. State: Replica: 0fec35079514420384128ceb783bbf34, State: Running, Role: LEADER
I20260926 05:41:46.169632 26875 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec35079514420384128ceb783bbf34" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40865 } }
I20260926 05:41:46.169689 26870 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:46.169965 26875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0fec35079514420384128ceb783bbf34. Latest consensus state: current_term: 1 leader_uuid: "0fec35079514420384128ceb783bbf34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec35079514420384128ceb783bbf34" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40865 } } }
I20260926 05:41:46.170054 26875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:46.169977 26876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0fec35079514420384128ceb783bbf34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0fec35079514420384128ceb783bbf34" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40865 } } }
I20260926 05:41:46.170365 26876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:46.171700 26880 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:46.172217 26880 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:46.173102 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:46.160552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45630 (local address 127.23.213.254:40865)
0926 05:41:46.160803 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:46.160814 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:46.161982 (+ 1168us) server_negotiation.cc:407] Connection header received
0926 05:41:46.162600 (+ 618us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:46.162609 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:46.162689 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:46.162829 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:46.163775 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.165183 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.166302 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.166855 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.166979 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:46.169699 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:46.169703 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:46.169703 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:46.169716 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:46.171512 (+ 1796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.171960 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.171963 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.171963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.172074 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.172304 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.172305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.172305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.172540 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:46.172701 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:46.172807 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 05:41:46.172942 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":161,"thread_start_us":97,"threads_started":1}
I20260926 05:41:46.174027 26880 catalog_manager.cc:1347] Generated new cluster ID: 8611a79933b94c689df580ac91fb118a
I20260926 05:41:46.174081 26880 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:46.186244 26880 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:46.186792 26880 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:46.189355 26880 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0fec35079514420384128ceb783bbf34: Generated new TSK 0
I20260926 05:41:46.189536 26880 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:46.224192 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40865
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:46.314553 26896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:46.314765 26896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:46.314788 26896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:46.315887 26896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:46.315932 26896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:46.315968 26896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:46.317371 26896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:46.317562 26896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:46.317788 26896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:46.322749 26904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:46.322772 26902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:46.322758 26901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:46.322909 26896 server_base.cc:1034] running on GCE node
I20260926 05:41:46.323195 26896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:46.323427 26896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:46.324577 26896 hybrid_clock.cc:648] HybridClock initialized: now 1790401306324573 us; error 34 us; skew 500 ppm
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:46.327057 26896 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:46.327471 26896 webserver.cc:460] Webserver started at http://127.23.213.193:45199/ using document root <none> and password file <none>
I20260926 05:41:46.327694 26896 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:46.327757 26896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:46.327903 26896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:46.328984 26896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "c7b8cc49e34d4a0dbafd1231dc283cb6"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "8376af22ca1a9137b3a5b5dde4f17312"
server_key_iv: "da5c790cb3405095d9ea71ed5044ea65"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.329357 26896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "c7b8cc49e34d4a0dbafd1231dc283cb6"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "8376af22ca1a9137b3a5b5dde4f17312"
server_key_iv: "da5c790cb3405095d9ea71ed5044ea65"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.330811 26896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:41:46.331485 26911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.331653 26896 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.331751 26896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "c7b8cc49e34d4a0dbafd1231dc283cb6"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "8376af22ca1a9137b3a5b5dde4f17312"
server_key_iv: "da5c790cb3405095d9ea71ed5044ea65"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.331820 26896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:46.345564 26896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:46.347425 26896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:46.347585 26896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:46.347823 26896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:46.348145 26896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:46.348194 26896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.348235 26896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:46.348263 26896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.353689 26896 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:46477
I20260926 05:41:46.353790 27024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:46477 every 8 connection(s)
I20260926 05:41:46.354092 26896 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:46.359467 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 26896
I20260926 05:41:46.359543 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:46.359730 24407 external_mini_cluster.cc:1428] Setting key a95c8508e030bb1d998f9ff7cedb5938
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:46.363325 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:46.354825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:56279 (local address 127.23.213.254:40865)
0926 05:41:46.354897 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:46.354899 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:46.355062 (+ 163us) server_negotiation.cc:407] Connection header received
0926 05:41:46.355526 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:46.355527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:46.355536 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:46.355584 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:46.356492 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.356978 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.358019 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.358229 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.358284 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:46.360470 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:46.360471 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:46.360471 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:46.360479 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:46.361955 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.362367 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.362369 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.362369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.362427 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.362677 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.362678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.362678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.362795 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:46.362868 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:46.363137 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 05:41:46.363236 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:41:46.363517 27027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:46.354777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40865 (local address 127.23.213.193:56279)
0926 05:41:46.354943 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:46.354978 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:46.355447 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:46.355652 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:46.355655 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:46.356217 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:46.356414 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:46.356419 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.357056 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.357056 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:46.357947 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:46.357950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.358301 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.358302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:46.358409 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:46.358863 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:46.358871 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:46.360363 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:46.362048 (+ 1685us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:46.362055 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:46.362058 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:46.362278 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:46.362491 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:46.362492 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:46.362492 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:46.362605 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:46.362860 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:46.362866 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:46.363006 (+ 140us) client_negotiation.cc:769] Sending connection context
0926 05:41:46.363115 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:41:46.363269 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":86,"threads_started":1}
I20260926 05:41:46.363983 27025 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40865
I20260926 05:41:46.364087 27025 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:46.364265 27025 heartbeater.cc:511] Master 127.23.213.254:40865 requested a full tablet report, sending...
I20260926 05:41:46.364768 26834 ts_manager.cc:194] Registered new tserver with Master: c7b8cc49e34d4a0dbafd1231dc283cb6 (127.23.213.193:46477)
I20260926 05:41:46.365381 26834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:56279
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:46.384167 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40865
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:46.463801 27030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:46.464005 27030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:46.464030 27030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:46.465137 27030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:46.465169 27030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:46.465195 27030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:46.466553 27030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:46.466753 27030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:46.466929 27030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:46.472149 27038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:46.472110 27036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:46.472193 27030 server_base.cc:1034] running on GCE node
W20260926 05:41:46.472126 27035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:46.472533 27030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:46.472806 27030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:46.473951 27030 hybrid_clock.cc:648] HybridClock initialized: now 1790401306473942 us; error 37 us; skew 500 ppm
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:46.476227 27030 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:46.476737 27030 webserver.cc:460] Webserver started at http://127.23.213.194:45539/ using document root <none> and password file <none>
I20260926 05:41:46.476970 27030 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:46.477034 27030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:46.477164 27030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:46.478287 27030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "af3a5123de684008bbb2533c3e25b69b"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "2d77ab7b8385867a1d9ba4b66d82fa46"
server_key_iv: "a1f6c94762c74c34673cf1fa3690f3f1"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.478644 27030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "af3a5123de684008bbb2533c3e25b69b"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "2d77ab7b8385867a1d9ba4b66d82fa46"
server_key_iv: "a1f6c94762c74c34673cf1fa3690f3f1"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.479921 27030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:46.480599 27045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.480762 27030 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:46.480854 27030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "af3a5123de684008bbb2533c3e25b69b"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "2d77ab7b8385867a1d9ba4b66d82fa46"
server_key_iv: "a1f6c94762c74c34673cf1fa3690f3f1"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.480931 27030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:46.510690 27030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:46.512512 27030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:46.512686 27030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:46.512923 27030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:46.513238 27030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:46.513288 27030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.513329 27030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:46.513357 27030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.519033 27030 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:32871
I20260926 05:41:46.519100 27158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:32871 every 8 connection(s)
I20260926 05:41:46.519395 27030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:46.527777 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:46.520021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:50299 (local address 127.23.213.254:40865)
0926 05:41:46.520102 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:46.520105 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:46.520288 (+ 183us) server_negotiation.cc:407] Connection header received
0926 05:41:46.520774 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:46.520775 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:46.520785 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:46.520839 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:46.521783 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.522282 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.523215 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.523393 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.523442 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:46.525454 (+ 2012us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:46.525455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:46.525456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:46.525461 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:46.526663 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.527002 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.527003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.527003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.527051 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.527253 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.527254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.527254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.527351 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:46.527399 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:46.527610 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 05:41:46.527680 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 05:41:46.527937 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:46.519987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40865 (local address 127.23.213.194:50299)
0926 05:41:46.520132 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:46.520177 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:46.520681 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:46.520893 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:46.520896 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:46.521465 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:46.521661 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:46.521665 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.522358 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.522359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:46.523139 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:46.523141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.523459 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.523460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:46.523559 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:46.524083 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:46.524091 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:46.525369 (+ 1278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:46.526730 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:46.526736 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:46.526737 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:46.526942 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:46.527103 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:46.527103 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:46.527104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:46.527204 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:46.527401 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:46.527405 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:46.527511 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 05:41:46.527589 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 05:41:46.527723 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":89,"threads_started":1}
I20260926 05:41:46.528450 27159 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40865
I20260926 05:41:46.528551 27159 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:46.528748 27159 heartbeater.cc:511] Master 127.23.213.254:40865 requested a full tablet report, sending...
I20260926 05:41:46.528790 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27030
I20260926 05:41:46.528891 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:46.529098 24407 external_mini_cluster.cc:1428] Setting key 075d8151a9afac5037b18e9c47a8d06c
I20260926 05:41:46.529155 26834 ts_manager.cc:194] Registered new tserver with Master: af3a5123de684008bbb2533c3e25b69b (127.23.213.194:32871)
I20260926 05:41:46.529604 26834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:50299
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:46.552416 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40865
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:46.632972 27164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:46.633265 27164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:46.633301 27164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:46.635088 27164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:46.635144 27164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:46.635183 27164 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:46.636875 27164 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40865
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:46.637117 27164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:46.637367 27164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:46.642330 27172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:46.642431 27164 server_base.cc:1034] running on GCE node
W20260926 05:41:46.642333 27170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:46.642334 27169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:46.642747 27164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:46.642951 27164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:46.644088 27164 hybrid_clock.cc:648] HybridClock initialized: now 1790401306644060 us; error 46 us; skew 500 ppm
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:46.646543 27164 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:46.646977 27164 webserver.cc:460] Webserver started at http://127.23.213.195:43181/ using document root <none> and password file <none>
I20260926 05:41:46.647194 27164 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:46.647238 27164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:46.647397 27164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:46.648473 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "d3008f0dd47b4ad590e144d73d92c714"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "56fc34e7cbe5c8941812de0d14b25336"
server_key_iv: "936ea9eac1a87e6ab5fa4a8bf428b76d"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.648845 27164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "d3008f0dd47b4ad590e144d73d92c714"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "56fc34e7cbe5c8941812de0d14b25336"
server_key_iv: "936ea9eac1a87e6ab5fa4a8bf428b76d"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.650192 27164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:46.650857 27179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.651019 27164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:46.651105 27164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "d3008f0dd47b4ad590e144d73d92c714"
format_stamp: "Formatted at 2026-09-26 05:41:46 on dist-test-slave-1fdt"
server_key: "56fc34e7cbe5c8941812de0d14b25336"
server_key_iv: "936ea9eac1a87e6ab5fa4a8bf428b76d"
server_key_version: "encryptionkey@0"
I20260926 05:41:46.651175 27164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:46.664740 27164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:46.667090 27164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:46.667204 27164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:46.667371 27164 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:46.667696 27164 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:46.667732 27164 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.667776 27164 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:46.667801 27164 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:46.674103 27164 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:40149
I20260926 05:41:46.674176 27292 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:40149 every 8 connection(s)
I20260926 05:41:46.674510 27164 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:46.676855 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27164
I20260926 05:41:46.676923 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:46.677111 24407 external_mini_cluster.cc:1428] Setting key 7cd61ecde1cfe2be3238f4273e98791c
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[26789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401306, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:46.683770 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:46.675268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:39993 (local address 127.23.213.254:40865)
0926 05:41:46.675329 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:46.675331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:46.675526 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:41:46.676027 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:46.676028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:46.676039 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:46.676082 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:46.677034 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.677824 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.678870 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.679056 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.679110 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:46.681143 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:46.681145 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:46.681145 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:46.681152 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:46.682412 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.682784 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.682785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.682785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.682833 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:46.683040 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:46.683040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:46.683041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:46.683199 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:46.683280 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:46.683597 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 05:41:46.683694 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:41:46.683956 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:46.675258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40865 (local address 127.23.213.195:39993)
0926 05:41:46.675403 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:46.675445 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:46.675928 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:46.676140 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:46.676143 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:46.676712 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:46.676944 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:46.676951 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.677933 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.677934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:46.678800 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:46.678803 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:46.679122 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:46.679123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:46.679221 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:46.679686 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:46.679694 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:46.681050 (+ 1356us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:46.682483 (+ 1433us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:46.682489 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:46.682491 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:46.682720 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:46.682888 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:46.682889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:46.682889 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:46.682991 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:46.683291 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:46.683297 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:46.683435 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 05:41:46.683528 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:41:46.683713 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260926 05:41:46.684454 27293 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40865
I20260926 05:41:46.684561 27293 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:46.684783 27293 heartbeater.cc:511] Master 127.23.213.254:40865 requested a full tablet report, sending...
I20260926 05:41:46.685180 26834 ts_manager.cc:194] Registered new tserver with Master: d3008f0dd47b4ad590e144d73d92c714 (127.23.213.195:40149)
I20260926 05:41:46.685554 26834 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:39993
I20260926 05:41:46.688810 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:41:46.699359 26873 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:46.698485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45638 (local address 127.23.213.254:40865)
0926 05:41:46.698563 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:46.698566 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:46.698645 (+ 79us) server_negotiation.cc:407] Connection header received
0926 05:41:46.698714 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:46.698715 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:46.698724 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:46.698774 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:46.699286 (+ 512us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45638: BlockingRecv error: recv got EOF from 127.0.0.1:45638 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:41:46.700158 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 26896
I20260926 05:41:46.704375 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27030
I20260926 05:41:46.708527 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27164
I20260926 05:41:46.712723 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 26802
2026-09-26T05:41:46Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1824 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[27305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[27305](info): set up 2 sockets
Sep 26 05:41:46 dist-test-slave-1fdt krb5kdc[27305](info): commencing operation
krb5kdc: starting...
W20260926 05:41:47.738971 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:47 dist-test-slave-1fdt krb5kdc[27305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:47.868013 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:37659
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:42735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:37659
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:47.948506 27318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:47.948709 27318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:47.948732 27318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:47.949896 27318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:47.949936 27318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:47.949950 27318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:47.949963 27318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:47.950022 27318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:47.951467 27318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:42735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:37659
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:37659
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:47.951720 27318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:47.951951 27318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:47.956799 27323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:47.956815 27318 server_base.cc:1034] running on GCE node
W20260926 05:41:47.956827 27324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:47.956799 27326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:47.957306 27318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:47.957558 27318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:47.958827 27318 hybrid_clock.cc:648] HybridClock initialized: now 1790401307958786 us; error 54 us; skew 500 ppm
Sep 26 05:41:47 dist-test-slave-1fdt krb5kdc[27305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401307, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:47.961187 27318 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:47.961566 27318 webserver.cc:460] Webserver started at http://127.23.213.254:35261/ using document root <none> and password file <none>
I20260926 05:41:47.961787 27318 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:47.961848 27318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:47.961946 27318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:47.963004 27318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "06d3237d006049799f3426807e7af183"
format_stamp: "Formatted at 2026-09-26 05:41:47 on dist-test-slave-1fdt"
server_key: "3d747f9703ed2abf333341ea9ec33ead"
server_key_iv: "7951daf412ae1a557d4fef9d1e227d32"
server_key_version: "encryptionkey@0"
I20260926 05:41:47.963310 27318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "06d3237d006049799f3426807e7af183"
format_stamp: "Formatted at 2026-09-26 05:41:47 on dist-test-slave-1fdt"
server_key: "3d747f9703ed2abf333341ea9ec33ead"
server_key_iv: "7951daf412ae1a557d4fef9d1e227d32"
server_key_version: "encryptionkey@0"
I20260926 05:41:47.964550 27318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:47.965186 27333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:47.965366 27318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:47.965452 27318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "06d3237d006049799f3426807e7af183"
format_stamp: "Formatted at 2026-09-26 05:41:47 on dist-test-slave-1fdt"
server_key: "3d747f9703ed2abf333341ea9ec33ead"
server_key_iv: "7951daf412ae1a557d4fef9d1e227d32"
server_key_version: "encryptionkey@0"
I20260926 05:41:47.965512 27318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:47.977805 27318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:47.979576 27318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:47.979684 27318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:47.983471 27318 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:37659
I20260926 05:41:47.983541 27385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:37659 every 8 connection(s)
I20260926 05:41:47.984006 27318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:47.984671 27386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:47.986980 27386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183: Bootstrap starting.
I20260926 05:41:47.987589 27386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:47.987888 27386 log.cc:824] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:47.988523 27386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183: No bootstrap required, opened a new log
I20260926 05:41:47.990091 27386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d3237d006049799f3426807e7af183" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37659 } }
I20260926 05:41:47.990226 27386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:47.990279 27386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06d3237d006049799f3426807e7af183, State: Initialized, Role: FOLLOWER
I20260926 05:41:47.990422 27386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d3237d006049799f3426807e7af183" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37659 } }
I20260926 05:41:47.990483 27386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:47.990522 27386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:47.990569 27386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:47.991400 27386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d3237d006049799f3426807e7af183" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37659 } }
I20260926 05:41:47.991505 27386 leader_election.cc:302] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 06d3237d006049799f3426807e7af183; no voters:
I20260926 05:41:47.991683 27386 leader_election.cc:288] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:47.991725 27389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:47.991894 27389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [term 1 LEADER]: Becoming Leader. State: Replica: 06d3237d006049799f3426807e7af183, State: Running, Role: LEADER
I20260926 05:41:47.991931 27386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:47.991976 27389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d3237d006049799f3426807e7af183" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37659 } }
I20260926 05:41:47.992327 27390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "06d3237d006049799f3426807e7af183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d3237d006049799f3426807e7af183" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37659 } } }
I20260926 05:41:47.992322 27391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 06d3237d006049799f3426807e7af183. Latest consensus state: current_term: 1 leader_uuid: "06d3237d006049799f3426807e7af183" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "06d3237d006049799f3426807e7af183" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37659 } } }
I20260926 05:41:47.992422 27390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:47.992429 27391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:47.992688 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27318
I20260926 05:41:47.992794 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:47.992995 24407 external_mini_cluster.cc:1428] Setting key 175e55bd29c7009519196bc0b4e91487
W20260926 05:41:47.993819 27405 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:41:47.993880 27405 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:41:47.993928 27396 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:47.994077 27396 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:47.995445 27396 catalog_manager.cc:1347] Generated new cluster ID: b4ccb08322bf463ca39ae63af7f8b847
I20260926 05:41:47.995496 27396 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:41:47 dist-test-slave-1fdt krb5kdc[27305](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401307, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:48.001550 27396 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:48.002125 27396 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:48.004953 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:47.993619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43326 (local address 127.23.213.254:37659)
0926 05:41:47.993896 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:47.993906 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:47.993939 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:41:47.994057 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:47.994064 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:47.994124 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:47.994266 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:47.995006 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:47.996393 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:47.997467 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:47.998103 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:47.998200 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:48.001027 (+ 2827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:48.001032 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:48.001032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:48.001046 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:48.003291 (+ 2245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:48.003721 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:48.003725 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:48.003725 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:48.003850 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:48.004051 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:48.004052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:48.004053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:48.004356 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:48.004524 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:48.004591 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 05:41:48.004742 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":224,"thread_start_us":90,"threads_started":1}
I20260926 05:41:48.005872 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:41:48.006808 27396 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 06d3237d006049799f3426807e7af183: Generated new TSK 0
I20260926 05:41:48.006969 27396 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:41:48.021392 27408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:48.017654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43328 (local address 127.23.213.254:37659)
0926 05:41:48.017703 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:48.017705 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:48.017828 (+ 123us) server_negotiation.cc:407] Connection header received
0926 05:41:48.017903 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:48.017904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:48.017917 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:48.017975 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:48.018309 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:48.019128 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:48.019840 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:48.020208 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:48.020272 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:48.020904 (+ 632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:48.020905 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:48.020905 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:48.021127 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:48.021154 (+ 27us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:48.021161 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:48.021226 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:48.021280 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 05:41:48.021307 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:41:48.023718 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27318
2026-09-26T05:41:48Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1309 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:48 dist-test-slave-1fdt krb5kdc[27419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:48 dist-test-slave-1fdt krb5kdc[27419](info): set up 2 sockets
Sep 26 05:41:48 dist-test-slave-1fdt krb5kdc[27419](info): commencing operation
krb5kdc: starting...
W20260926 05:41:49.053864 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:49.167990 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:39741
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:39741
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:49.247279 27432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:49.247514 27432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:49.247550 27432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:49.248699 27432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:49.248762 27432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:49.248786 27432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:49.248816 27432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:49.248840 27432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:49.250339 27432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:39741
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:39741
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:49.250613 27432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:49.250829 27432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:49.255498 27438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:49.255566 27437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:49.255692 27432 server_base.cc:1034] running on GCE node
W20260926 05:41:49.255507 27440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:49.256031 27432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:49.256312 27432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:49.257474 27432 hybrid_clock.cc:648] HybridClock initialized: now 1790401309257461 us; error 30 us; skew 500 ppm
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:49.259923 27432 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:49.260345 27432 webserver.cc:460] Webserver started at http://127.23.213.254:40637/ using document root <none> and password file <none>
I20260926 05:41:49.260562 27432 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:49.260624 27432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:49.260776 27432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:49.261894 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "2ff18cde51244233b7d222cd4a394259"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "47b59cda0c714af094c243ce0aa52f48"
server_key_iv: "805726e658d2d15b79afe63a2dc95042"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.262274 27432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "2ff18cde51244233b7d222cd4a394259"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "47b59cda0c714af094c243ce0aa52f48"
server_key_iv: "805726e658d2d15b79afe63a2dc95042"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.263566 27432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:49.264283 27447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.264447 27432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:49.264547 27432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "2ff18cde51244233b7d222cd4a394259"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "47b59cda0c714af094c243ce0aa52f48"
server_key_iv: "805726e658d2d15b79afe63a2dc95042"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.264617 27432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:49.273236 27432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:49.275076 27432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:49.275234 27432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:49.279078 27432 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:39741
I20260926 05:41:49.279114 27499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:39741 every 8 connection(s)
I20260926 05:41:49.279449 27432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:49.280408 27500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:49.282478 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27432
I20260926 05:41:49.282591 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:49.282799 24407 external_mini_cluster.cc:1428] Setting key 6d9fb6f0265b60dabee869e4208f0562
I20260926 05:41:49.282883 27500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259: Bootstrap starting.
I20260926 05:41:49.283535 27500 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:49.283824 27500 log.cc:824] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:49.284446 27500 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259: No bootstrap required, opened a new log
I20260926 05:41:49.286274 27500 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff18cde51244233b7d222cd4a394259" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39741 } }
I20260926 05:41:49.286418 27500 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:49.286463 27500 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ff18cde51244233b7d222cd4a394259, State: Initialized, Role: FOLLOWER
I20260926 05:41:49.286619 27500 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff18cde51244233b7d222cd4a394259" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39741 } }
I20260926 05:41:49.286717 27500 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:49.286761 27500 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:49.286810 27500 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:49.287321 27500 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff18cde51244233b7d222cd4a394259" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39741 } }
I20260926 05:41:49.287431 27500 leader_election.cc:302] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2ff18cde51244233b7d222cd4a394259; no voters:
I20260926 05:41:49.287593 27500 leader_election.cc:288] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:49.287629 27505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:49.287847 27500 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:49.288759 27505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [term 1 LEADER]: Becoming Leader. State: Replica: 2ff18cde51244233b7d222cd4a394259, State: Running, Role: LEADER
I20260926 05:41:49.288853 27505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff18cde51244233b7d222cd4a394259" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39741 } }
I20260926 05:41:49.289199 27512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ff18cde51244233b7d222cd4a394259. Latest consensus state: current_term: 1 leader_uuid: "2ff18cde51244233b7d222cd4a394259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff18cde51244233b7d222cd4a394259" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39741 } } }
I20260926 05:41:49.289278 27512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [sys.catalog]: This master's current role is: LEADER
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:49.289382 27506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2ff18cde51244233b7d222cd4a394259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2ff18cde51244233b7d222cd4a394259" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39741 } } }
I20260926 05:41:49.289635 27506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:49.289539 27520 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:49.290295 27520 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:49.292122 27520 catalog_manager.cc:1347] Generated new cluster ID: 49978fa8736a4eb7843e6921fbd2c56e
I20260926 05:41:49.292173 27520 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:49.293655 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.283329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.23.213.254:39741)
0926 05:41:49.283513 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:49.283520 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:49.283544 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:41:49.284061 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:49.284068 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:49.284116 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:49.284259 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:49.284920 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.285821 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.286644 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.287151 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.287265 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.289941 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:49.289944 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:49.289945 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:49.289957 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:49.291958 (+ 2001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.292383 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.292387 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.292387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.292501 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.292712 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.292713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.292714 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.293022 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:49.293170 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:49.293285 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 05:41:49.293432 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260926 05:41:49.299021 27520 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:49.299500 27520 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:49.307037 27520 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2ff18cde51244233b7d222cd4a394259: Generated new TSK 0
I20260926 05:41:49.307207 27520 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:49.340732 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39741
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:49.436967 27526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:49.437196 27526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:49.437237 27526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:49.438458 27526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:49.438519 27526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:49.438565 27526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:49.440013 27526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:49.440271 27526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:49.440500 27526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:49.445467 27531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:49.445549 27534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:49.445473 27532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:49.445709 27526 server_base.cc:1034] running on GCE node
I20260926 05:41:49.445941 27526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:49.446234 27526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:49.447396 27526 hybrid_clock.cc:648] HybridClock initialized: now 1790401309447384 us; error 41 us; skew 500 ppm
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:49.449734 27526 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:49.450187 27526 webserver.cc:460] Webserver started at http://127.23.213.193:35133/ using document root <none> and password file <none>
I20260926 05:41:49.450361 27526 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:49.450400 27526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:49.450486 27526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:49.451463 27526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "d59de7eeb186404d9f92cc4e00ecd988"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "56064caf85bbf85427abb22adcbc904f"
server_key_iv: "202c82c32467791ea51d01b8f9d2e9b6"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.451756 27526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "d59de7eeb186404d9f92cc4e00ecd988"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "56064caf85bbf85427abb22adcbc904f"
server_key_iv: "202c82c32467791ea51d01b8f9d2e9b6"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.452955 27526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:49.453608 27541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.453775 27526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:41:49.453860 27526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "d59de7eeb186404d9f92cc4e00ecd988"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "56064caf85bbf85427abb22adcbc904f"
server_key_iv: "202c82c32467791ea51d01b8f9d2e9b6"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.453913 27526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:49.479847 27526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:49.482263 27526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:49.482380 27526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:49.482582 27526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:49.482915 27526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:49.482951 27526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.482975 27526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:49.483021 27526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.488941 27526 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:42627
I20260926 05:41:49.489037 27654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:42627 every 8 connection(s)
I20260926 05:41:49.489354 27526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:49.496562 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27526
I20260926 05:41:49.496704 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:49.496987 24407 external_mini_cluster.cc:1428] Setting key 7c2c6685af91d27e0d819800f696ba65
I20260926 05:41:49.499583 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.490091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:42389 (local address 127.23.213.254:39741)
0926 05:41:49.490149 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:49.490151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:49.490419 (+ 268us) server_negotiation.cc:407] Connection header received
0926 05:41:49.490946 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:49.490948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:49.490959 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:49.491014 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:49.491942 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.492464 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.493444 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.493690 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.493848 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.496073 (+ 2225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:49.496074 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:49.496074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:49.496081 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:49.497897 (+ 1816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.498459 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.498460 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.498460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.498510 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.498808 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.498809 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.498809 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.498906 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:49.498954 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:49.499352 (+ 398us) server_negotiation.cc:299] Negotiation successful
0926 05:41:49.499457 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:41:49.499835 27657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.490098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39741 (local address 127.23.213.193:42389)
0926 05:41:49.490263 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:49.490314 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:49.490821 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:49.491069 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:49.491073 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:49.491626 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:49.491828 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:49.491833 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.492548 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.492548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:49.493349 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:49.493351 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.493857 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.493858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:49.493961 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.494451 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:49.494458 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:49.495987 (+ 1529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:49.497988 (+ 2001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:49.497996 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:49.497999 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:49.498375 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:49.498576 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:49.498576 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:49.498577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:49.498740 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:49.498981 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:49.498988 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:49.499179 (+ 191us) client_negotiation.cc:769] Sending connection context
0926 05:41:49.499318 (+ 139us) client_negotiation.cc:240] Negotiation successful
0926 05:41:49.499524 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":94,"threads_started":1}
I20260926 05:41:49.500388 27655 heartbeater.cc:347] Connected to a master server at 127.23.213.254:39741
I20260926 05:41:49.500515 27655 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:49.500712 27655 heartbeater.cc:511] Master 127.23.213.254:39741 requested a full tablet report, sending...
I20260926 05:41:49.501293 27464 ts_manager.cc:194] Registered new tserver with Master: d59de7eeb186404d9f92cc4e00ecd988 (127.23.213.193:42627)
I20260926 05:41:49.502110 27464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:42389
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:49.520892 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39741
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:49.601568 27660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:49.601840 27660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:49.601871 27660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:49.603304 27660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:49.603356 27660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:49.603385 27660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:49.605026 27660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27660
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:49.605269 27660 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:49.605504 27660 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:49.610500 27666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:49.610476 27665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:49.610618 27660 server_base.cc:1034] running on GCE node
W20260926 05:41:49.610458 27668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:49.610885 27660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:49.611104 27660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:49.612252 27660 hybrid_clock.cc:648] HybridClock initialized: now 1790401309612239 us; error 28 us; skew 500 ppm
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:49.614728 27660 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:49.615206 27660 webserver.cc:460] Webserver started at http://127.23.213.194:40227/ using document root <none> and password file <none>
I20260926 05:41:49.615448 27660 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:49.615494 27660 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:49.615655 27660 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:49.616745 27660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "1f163f1de1064c3280acad9257c5cb78"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "52b7e69e09424d69e7fc820535f2f5fe"
server_key_iv: "bc9ee0cf3f7f48ff644e31a14a3d574d"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.617090 27660 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "1f163f1de1064c3280acad9257c5cb78"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "52b7e69e09424d69e7fc820535f2f5fe"
server_key_iv: "bc9ee0cf3f7f48ff644e31a14a3d574d"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.618533 27660 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:49.619323 27675 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.619496 27660 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:49.619592 27660 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "1f163f1de1064c3280acad9257c5cb78"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "52b7e69e09424d69e7fc820535f2f5fe"
server_key_iv: "bc9ee0cf3f7f48ff644e31a14a3d574d"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.619676 27660 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:49.635428 27660 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:49.637230 27660 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:49.637384 27660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:49.637598 27660 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:49.637980 27660 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:49.638015 27660 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.638036 27660 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:49.638079 27660 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.644011 27660 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:35675
I20260926 05:41:49.644078 27788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:35675 every 8 connection(s)
I20260926 05:41:49.644465 27660 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:41:49.645475 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27660
I20260926 05:41:49.645601 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:49.645835 24407 external_mini_cluster.cc:1428] Setting key 789dccb423686743cdd6a82f1fd8dfd4
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
I20260926 05:41:49.654876 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.645237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:34873 (local address 127.23.213.254:39741)
0926 05:41:49.645359 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:49.645362 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:49.645550 (+ 188us) server_negotiation.cc:407] Connection header received
0926 05:41:49.646345 (+ 795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:49.646347 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:49.646358 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:49.646409 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:49.647726 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.648455 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.649459 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.649681 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.649786 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.651963 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:49.651965 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:49.651965 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:49.651973 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:49.653442 (+ 1469us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.653953 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.653954 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.653954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.654013 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.654244 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.654245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.654245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.654359 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:49.654427 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:49.654709 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 05:41:49.654786 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260926 05:41:49.655053 27791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.645222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39741 (local address 127.23.213.194:34873)
0926 05:41:49.645404 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:49.645451 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:49.646237 (+ 786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:49.646533 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:49.646537 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:49.647324 (+ 787us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:49.647592 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:49.647598 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.648551 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.648552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:49.649366 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:49.649368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.649789 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.649790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:49.649920 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.650533 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:49.650541 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:49.651869 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:49.653568 (+ 1699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:49.653574 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:49.653577 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:49.653877 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:49.654066 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:49.654067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:49.654067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:49.654174 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:49.654433 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:49.654447 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:49.654578 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 05:41:49.654673 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:41:49.654826 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":99,"threads_started":1}
I20260926 05:41:49.655467 27789 heartbeater.cc:347] Connected to a master server at 127.23.213.254:39741
I20260926 05:41:49.655601 27789 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:49.655829 27789 heartbeater.cc:511] Master 127.23.213.254:39741 requested a full tablet report, sending...
I20260926 05:41:49.656241 27464 ts_manager.cc:194] Registered new tserver with Master: 1f163f1de1064c3280acad9257c5cb78 (127.23.213.194:35675)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
I20260926 05:41:49.656755 27464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:34873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:49.672685 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39741
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:49.752213 27794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:49.752434 27794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:49.752471 27794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:49.753608 27794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:49.753671 27794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:49.753715 27794 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:49.755167 27794 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39741
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:49.755416 27794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:49.755635 27794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:49.760553 27799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:49.760571 27802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:49.760759 27794 server_base.cc:1034] running on GCE node
W20260926 05:41:49.760555 27800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:49.761037 27794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:49.761288 27794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:49.762425 27794 hybrid_clock.cc:648] HybridClock initialized: now 1790401309762414 us; error 28 us; skew 500 ppm
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:49.764958 27794 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:49.765344 27794 webserver.cc:460] Webserver started at http://127.23.213.195:44127/ using document root <none> and password file <none>
I20260926 05:41:49.765547 27794 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:49.765590 27794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:49.765681 27794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:49.766729 27794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "f2b93564c083488ca5b158809377e1dc"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "9c39bed791723c835f68536a8f046f82"
server_key_iv: "83fd765145624e6edf77edb0867f0f03"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.767015 27794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "f2b93564c083488ca5b158809377e1dc"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "9c39bed791723c835f68536a8f046f82"
server_key_iv: "83fd765145624e6edf77edb0867f0f03"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.768440 27794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:49.769169 27809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.769323 27794 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:49.769410 27794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "f2b93564c083488ca5b158809377e1dc"
format_stamp: "Formatted at 2026-09-26 05:41:49 on dist-test-slave-1fdt"
server_key: "9c39bed791723c835f68536a8f046f82"
server_key_iv: "83fd765145624e6edf77edb0867f0f03"
server_key_version: "encryptionkey@0"
I20260926 05:41:49.769469 27794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:49.797250 27794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:49.799520 27794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:49.799641 27794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:49.799849 27794 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:49.800212 27794 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:49.800249 27794 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.800295 27794 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:49.800323 27794 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:49.806170 27794 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:35215
I20260926 05:41:49.806227 27922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:35215 every 8 connection(s)
I20260926 05:41:49.806574 27794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:49.807873 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27794
I20260926 05:41:49.807958 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:49.808142 24407 external_mini_cluster.cc:1428] Setting key b61394fdbb5816a975427940a52e45a8
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:49.816004 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.807312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:47481 (local address 127.23.213.254:39741)
0926 05:41:49.807389 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:49.807392 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:49.807583 (+ 191us) server_negotiation.cc:407] Connection header received
0926 05:41:49.808293 (+ 710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:49.808294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:49.808306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:49.808367 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:49.809347 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.809917 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.810947 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.811156 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.811240 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.813357 (+ 2117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:49.813359 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:49.813359 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:49.813367 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:49.814634 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.815049 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.815050 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.815051 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.815109 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.815356 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.815357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.815357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.815475 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:49.815544 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:49.815826 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 05:41:49.815908 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:41:49.816195 27925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.807301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39741 (local address 127.23.213.195:47481)
0926 05:41:49.807465 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:49.807506 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:49.808196 (+ 690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:49.808445 (+ 249us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:49.808449 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:49.809046 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:49.809247 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:49.809253 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.810005 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.810006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:49.810854 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:49.810856 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.811249 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.811250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:49.811391 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.811917 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:49.811926 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:49.813260 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:49.814731 (+ 1471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:49.814738 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:49.814740 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:49.814963 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:49.815176 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:49.815177 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:49.815177 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:49.815284 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:49.815549 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:49.815554 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:49.815699 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 05:41:49.815805 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 05:41:49.815964 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":102,"threads_started":1}
I20260926 05:41:49.816658 27923 heartbeater.cc:347] Connected to a master server at 127.23.213.254:39741
I20260926 05:41:49.816749 27923 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:49.816895 27923 heartbeater.cc:511] Master 127.23.213.254:39741 requested a full tablet report, sending...
I20260926 05:41:49.817233 27464 ts_manager.cc:194] Registered new tserver with Master: f2b93564c083488ca5b158809377e1dc (127.23.213.195:35215)
I20260926 05:41:49.817601 27464 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:47481
I20260926 05:41:49.819567 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401309, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:49.841557 27503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:49.834945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49056 (local address 127.23.213.254:39741)
0926 05:41:49.835035 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:49.835037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:49.835089 (+ 52us) server_negotiation.cc:407] Connection header received
0926 05:41:49.835161 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:49.835162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:49.835171 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:49.835222 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:49.835772 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.836257 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.836936 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:49.837109 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:49.837158 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:49.839405 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:49.839406 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:49.839407 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:49.839414 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:49.840580 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.840886 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.840887 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.840887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.840938 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:49.841091 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:49.841092 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:49.841092 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:49.841233 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:49.841309 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:49.841386 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 05:41:49.841448 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
W20260926 05:41:49.841856 27464 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49056
I20260926 05:41:49.842339 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27526
I20260926 05:41:49.846999 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27660
I20260926 05:41:49.851646 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27794
I20260926 05:41:49.855644 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27432
2026-09-26T05:41:49Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1833 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27935](info): set up 2 sockets
Sep 26 05:41:49 dist-test-slave-1fdt krb5kdc[27935](info): commencing operation
krb5kdc: starting...
W20260926 05:41:50.885576 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:50 dist-test-slave-1fdt krb5kdc[27935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:50.999830 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40421
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:40421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:51.081178 27948 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:51.081380 27948 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:51.081403 27948 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:51.082590 27948 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:51.082634 27948 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:51.082649 27948 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:51.082684 27948 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:51.082712 27948 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:51.084304 27948 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:40421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27948
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:51.084551 27948 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:51.084771 27948 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:41:51.089529 27948 server_base.cc:1034] running on GCE node
W20260926 05:41:51.089506 27953 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:51.089486 27956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:51.089495 27954 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:51.089962 27948 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:51.090242 27948 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:51.091405 27948 hybrid_clock.cc:648] HybridClock initialized: now 1790401311091390 us; error 26 us; skew 500 ppm
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:51.100016 27948 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:51.100520 27948 webserver.cc:460] Webserver started at http://127.23.213.254:33393/ using document root <none> and password file <none>
I20260926 05:41:51.100757 27948 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:51.100822 27948 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:51.100968 27948 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:51.102118 27948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "4ab23db72eab4a828e1bf133af348537"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "791881636799743dee0301a3414119b4"
server_key_iv: "a8212bda978ffbd54141d54d339c42c7"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.102486 27948 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "4ab23db72eab4a828e1bf133af348537"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "791881636799743dee0301a3414119b4"
server_key_iv: "a8212bda978ffbd54141d54d339c42c7"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.103868 27948 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:51.104579 27963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.104750 27948 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:51.104858 27948 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "4ab23db72eab4a828e1bf133af348537"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "791881636799743dee0301a3414119b4"
server_key_iv: "a8212bda978ffbd54141d54d339c42c7"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.104930 27948 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:51.114820 27948 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:51.116609 27948 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:51.116794 27948 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:51.120905 27948 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:40421
I20260926 05:41:51.120942 28015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:40421 every 8 connection(s)
I20260926 05:41:51.121445 27948 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:51.122148 28016 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:51.124396 28016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537: Bootstrap starting.
I20260926 05:41:51.124544 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 27948
I20260926 05:41:51.124663 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:51.124830 24407 external_mini_cluster.cc:1428] Setting key 5332ab494db35e17c4292b896b6b339e
I20260926 05:41:51.125145 28016 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:51.125507 28016 log.cc:824] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:51.126302 28016 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537: No bootstrap required, opened a new log
I20260926 05:41:51.128599 28016 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab23db72eab4a828e1bf133af348537" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40421 } }
I20260926 05:41:51.128747 28016 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:51.128770 28016 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ab23db72eab4a828e1bf133af348537, State: Initialized, Role: FOLLOWER
I20260926 05:41:51.128906 28016 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab23db72eab4a828e1bf133af348537" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40421 } }
I20260926 05:41:51.128986 28016 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:51.129034 28016 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:51.129081 28016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:51.129704 28016 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab23db72eab4a828e1bf133af348537" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40421 } }
I20260926 05:41:51.129822 28016 leader_election.cc:302] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ab23db72eab4a828e1bf133af348537; no voters:
I20260926 05:41:51.129969 28016 leader_election.cc:288] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:51.130021 28021 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:51.130213 28016 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:51.130193 28021 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [term 1 LEADER]: Becoming Leader. State: Replica: 4ab23db72eab4a828e1bf133af348537, State: Running, Role: LEADER
I20260926 05:41:51.130357 28021 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab23db72eab4a828e1bf133af348537" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40421 } }
I20260926 05:41:51.130685 28021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ab23db72eab4a828e1bf133af348537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab23db72eab4a828e1bf133af348537" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40421 } } }
I20260926 05:41:51.130764 28021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:51.130897 28021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ab23db72eab4a828e1bf133af348537. Latest consensus state: current_term: 1 leader_uuid: "4ab23db72eab4a828e1bf133af348537" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ab23db72eab4a828e1bf133af348537" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40421 } } }
I20260926 05:41:51.130947 28021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:51.131103 28028 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:51.131886 28028 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:51.136376 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.125353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60622 (local address 127.23.213.254:40421)
0926 05:41:51.125552 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:51.125563 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:51.125594 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:41:51.126190 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:51.126200 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:51.126261 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:51.126411 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:51.127027 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.128481 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.129438 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.130102 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.130217 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.132998 (+ 2781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:51.133001 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:51.133001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:51.133013 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:51.134886 (+ 1873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.135211 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.135214 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.135214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.135321 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.135499 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.135500 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.135500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.135810 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:51.135947 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:51.136025 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:41:51.136159 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":90,"threads_started":1}
I20260926 05:41:51.137523 28028 catalog_manager.cc:1347] Generated new cluster ID: 4a78a3155f7d4037838e259bfd471706
I20260926 05:41:51.137579 28028 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:51.154481 28028 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:51.155222 28028 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:51.168008 28028 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ab23db72eab4a828e1bf133af348537: Generated new TSK 0
I20260926 05:41:51.168227 28028 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:51.198083 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40421
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:51.278779 28042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:51.278981 28042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:51.279006 28042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:51.280200 28042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:51.280247 28042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:51.280304 28042 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:51.281864 28042 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:51.282117 28042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:51.282351 28042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:51.287370 28050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:51.287346 28047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:51.287355 28048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:51.287451 28042 server_base.cc:1034] running on GCE node
I20260926 05:41:51.287709 28042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:51.287954 28042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:51.289115 28042 hybrid_clock.cc:648] HybridClock initialized: now 1790401311289103 us; error 42 us; skew 500 ppm
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:51.291742 28042 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:51.292176 28042 webserver.cc:460] Webserver started at http://127.23.213.193:46791/ using document root <none> and password file <none>
I20260926 05:41:51.292452 28042 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:51.292514 28042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:51.292647 28042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:51.293645 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "b3def9e807eb4c4fa0e8b68638af78ec"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "a545a0fc2e25f2326a4ed4ef594c79a4"
server_key_iv: "925a850a5c330f61b5924b0f5c3e9e9a"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.294026 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "b3def9e807eb4c4fa0e8b68638af78ec"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "a545a0fc2e25f2326a4ed4ef594c79a4"
server_key_iv: "925a850a5c330f61b5924b0f5c3e9e9a"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.295331 28042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:51.295984 28057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.296149 28042 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:51.296237 28042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "b3def9e807eb4c4fa0e8b68638af78ec"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "a545a0fc2e25f2326a4ed4ef594c79a4"
server_key_iv: "925a850a5c330f61b5924b0f5c3e9e9a"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.296316 28042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:51.349324 28042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:51.351301 28042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:51.351456 28042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:51.351694 28042 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:51.351995 28042 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:51.352044 28042 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.352087 28042 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:51.352126 28042 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.357642 28042 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:38189
I20260926 05:41:51.357766 28170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:38189 every 8 connection(s)
I20260926 05:41:51.358064 28042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:51.363308 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28042
I20260926 05:41:51.363389 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:51.363554 24407 external_mini_cluster.cc:1428] Setting key 8f6f8ad6040fd8184064fec57366538e
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:51.367753 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.358796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:56875 (local address 127.23.213.254:40421)
0926 05:41:51.358869 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:51.358872 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:51.359038 (+ 166us) server_negotiation.cc:407] Connection header received
0926 05:41:51.359943 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:51.359944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:51.359954 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:51.359995 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:51.360579 (+ 584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.361129 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.362087 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.362293 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.362344 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.364628 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:51.364629 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:51.364629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:51.364637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:51.366188 (+ 1551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.366680 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.366681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.366681 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.366742 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.367016 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.367017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.367017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.367176 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:51.367245 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:51.367578 (+ 333us) server_negotiation.cc:299] Negotiation successful
0926 05:41:51.367664 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:41:51.368213 28173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.358752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40421 (local address 127.23.213.193:56875)
0926 05:41:51.358924 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:51.358960 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:51.359846 (+ 886us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:51.360052 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:51.360055 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:51.360314 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:51.360507 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:51.360512 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.361205 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.361206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:51.362025 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:51.362027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.362367 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.362368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:51.362478 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.363000 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:51.363008 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:51.364541 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:51.366265 (+ 1724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:51.366272 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:51.366274 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:51.366600 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:51.366818 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:51.366818 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:51.366819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:51.366948 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:51.367242 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:51.367248 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:51.367419 (+ 171us) client_negotiation.cc:769] Sending connection context
0926 05:41:51.367862 (+ 443us) client_negotiation.cc:240] Negotiation successful
0926 05:41:51.368047 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260926 05:41:51.368669 28171 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40421
I20260926 05:41:51.368762 28171 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:51.368950 28171 heartbeater.cc:511] Master 127.23.213.254:40421 requested a full tablet report, sending...
I20260926 05:41:51.369443 27980 ts_manager.cc:194] Registered new tserver with Master: b3def9e807eb4c4fa0e8b68638af78ec (127.23.213.193:38189)
I20260926 05:41:51.370078 27980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:56875
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:51.386636 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40421
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:51.466090 28176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:51.466292 28176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:51.466316 28176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:51.467391 28176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:51.467422 28176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:51.467446 28176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:51.468823 28176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28176
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:51.468992 28176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:51.469165 28176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:51.473935 28181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:51.474053 28176 server_base.cc:1034] running on GCE node
W20260926 05:41:51.473948 28184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:51.474092 28182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:51.474359 28176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:51.474562 28176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:51.475696 28176 hybrid_clock.cc:648] HybridClock initialized: now 1790401311475672 us; error 40 us; skew 500 ppm
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:51.478219 28176 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:51.478590 28176 webserver.cc:460] Webserver started at http://127.23.213.194:35381/ using document root <none> and password file <none>
I20260926 05:41:51.478772 28176 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:51.478812 28176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:51.478896 28176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:51.479936 28176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "a971e8e7866f4ad0801ceb123bc78076"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "feca65063393bc884ce685ff0c56ba0f"
server_key_iv: "588dee362eebce1506ba323f55150247"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.480233 28176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "a971e8e7866f4ad0801ceb123bc78076"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "feca65063393bc884ce685ff0c56ba0f"
server_key_iv: "588dee362eebce1506ba323f55150247"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.481501 28176 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:41:51.482234 28191 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.482391 28176 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.482491 28176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "a971e8e7866f4ad0801ceb123bc78076"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "feca65063393bc884ce685ff0c56ba0f"
server_key_iv: "588dee362eebce1506ba323f55150247"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.482561 28176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:51.501719 28176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:51.504143 28176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:51.504303 28176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:51.504541 28176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:51.504850 28176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:51.504900 28176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.504940 28176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:51.504971 28176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.511394 28176 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:44289
I20260926 05:41:51.511444 28304 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:44289 every 8 connection(s)
I20260926 05:41:51.511760 28176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:51.520483 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.512436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:53033 (local address 127.23.213.254:40421)
0926 05:41:51.512501 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:51.512503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:51.512722 (+ 219us) server_negotiation.cc:407] Connection header received
0926 05:41:51.513596 (+ 874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:51.513597 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:51.513607 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:51.513663 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:51.514301 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.514788 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.515713 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.515893 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.515942 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.517982 (+ 2040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:51.517984 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:51.517984 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:51.517992 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:51.519140 (+ 1148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.519542 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.519543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.519543 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.519600 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.519843 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.519844 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.519845 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.519963 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:51.520030 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:51.520309 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 05:41:51.520389 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:41:51.520677 28307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.512429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40421 (local address 127.23.213.194:53033)
0926 05:41:51.512595 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:51.512632 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:51.513509 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:51.513716 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:51.513719 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:51.514030 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:51.514227 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:51.514232 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.514859 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.514860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:51.515640 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:51.515642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.515956 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.515957 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:51.516052 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.516488 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:51.516496 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:51.517888 (+ 1392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:51.519234 (+ 1346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:51.519240 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:51.519242 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:51.519459 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:51.519668 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:51.519668 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:51.519669 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:51.519778 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:51.520033 (+ 255us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:51.520040 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:51.520178 (+ 138us) client_negotiation.cc:769] Sending connection context
0926 05:41:51.520287 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:41:51.520444 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
I20260926 05:41:51.521159 28305 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40421
I20260926 05:41:51.521209 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28176
I20260926 05:41:51.521265 28305 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:51.521323 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:51.521445 28305 heartbeater.cc:511] Master 127.23.213.254:40421 requested a full tablet report, sending...
I20260926 05:41:51.521495 24407 external_mini_cluster.cc:1428] Setting key d4e04f2c19b996a266ccafd5267c9025
I20260926 05:41:51.521869 27980 ts_manager.cc:194] Registered new tserver with Master: a971e8e7866f4ad0801ceb123bc78076 (127.23.213.194:44289)
I20260926 05:41:51.522392 27980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:53033
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:51.544391 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40421
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:51.624990 28310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:51.625185 28310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:51.625208 28310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:51.626438 28310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:51.626480 28310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:51.626507 28310 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:51.627934 28310 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:51.628108 28310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:51.628280 28310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:51.633435 28316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:51.633440 28318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:51.633487 28310 server_base.cc:1034] running on GCE node
W20260926 05:41:51.633455 28315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:51.633874 28310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:51.634130 28310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:51.635268 28310 hybrid_clock.cc:648] HybridClock initialized: now 1790401311635263 us; error 33 us; skew 500 ppm
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:51.637547 28310 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:51.638022 28310 webserver.cc:460] Webserver started at http://127.23.213.195:36689/ using document root <none> and password file <none>
I20260926 05:41:51.638260 28310 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:51.638327 28310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:51.638468 28310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:51.639508 28310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "9ebbed1e0b184e3c8d43ff0156864fcd"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "50558ebf3aafe735380b13c9331e052a"
server_key_iv: "23c0f3ea892b41eab73c229ec4b013e7"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.639855 28310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "9ebbed1e0b184e3c8d43ff0156864fcd"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "50558ebf3aafe735380b13c9331e052a"
server_key_iv: "23c0f3ea892b41eab73c229ec4b013e7"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.641119 28310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:41:51.641848 28325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.642014 28310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.642110 28310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "9ebbed1e0b184e3c8d43ff0156864fcd"
format_stamp: "Formatted at 2026-09-26 05:41:51 on dist-test-slave-1fdt"
server_key: "50558ebf3aafe735380b13c9331e052a"
server_key_iv: "23c0f3ea892b41eab73c229ec4b013e7"
server_key_version: "encryptionkey@0"
I20260926 05:41:51.642189 28310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:51.660133 28310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:51.661923 28310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:51.662070 28310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:51.662302 28310 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:51.662603 28310 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:51.662652 28310 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.662693 28310 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:51.662721 28310 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:51.668380 28310 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:34363
I20260926 05:41:51.668447 28438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:34363 every 8 connection(s)
I20260926 05:41:51.668751 28310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:51.669224 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28310
I20260926 05:41:51.669303 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:51.669514 24407 external_mini_cluster.cc:1428] Setting key 7a7fa4951085cd1f122139e319342f00
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:51.677726 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.669517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:42353 (local address 127.23.213.254:40421)
0926 05:41:51.669581 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:51.669583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:51.670000 (+ 417us) server_negotiation.cc:407] Connection header received
0926 05:41:51.671089 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:51.671090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:51.671101 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:51.671150 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:51.671739 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.672220 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.673146 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.673328 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.673374 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.675329 (+ 1955us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:51.675330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:51.675330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:51.675336 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:51.676681 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.677002 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.677003 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.677003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.677049 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.677242 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.677243 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.677243 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.677339 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:51.677387 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:51.677605 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 05:41:51.677663 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:41:51.678031 28441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.669507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40421 (local address 127.23.213.195:42353)
0926 05:41:51.669885 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:51.669933 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:51.671003 (+ 1070us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:51.671202 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:51.671205 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:51.671452 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:51.671671 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:51.671675 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.672290 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.672290 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:51.673071 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:51.673073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.673391 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.673391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:51.673486 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.673954 (+ 468us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:51.673962 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:51.675253 (+ 1291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:51.676745 (+ 1492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:51.676749 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:51.676751 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:51.676945 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:51.677101 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:51.677101 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:51.677102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:51.677193 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:51.677389 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:51.677392 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:51.677498 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 05:41:51.677576 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 05:41:51.677711 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":275,"thread_start_us":95,"threads_started":1}
I20260926 05:41:51.678561 28439 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40421
I20260926 05:41:51.678676 28439 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:51.678851 28439 heartbeater.cc:511] Master 127.23.213.254:40421 requested a full tablet report, sending...
I20260926 05:41:51.679177 27980 ts_manager.cc:194] Registered new tserver with Master: 9ebbed1e0b184e3c8d43ff0156864fcd (127.23.213.195:34363)
I20260926 05:41:51.679500 27980 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:42353
I20260926 05:41:51.681182 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:51.703387 28446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.695684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59718 (local address 127.23.213.193:38189)
0926 05:41:51.695846 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:51.695852 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:51.695862 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:41:51.695895 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:51.695897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:51.695910 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:51.695988 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:51.696579 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.697311 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.698112 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.698478 (+ 366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.698566 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.700769 (+ 2203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:51.700775 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:51.700776 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:51.700782 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:51.702166 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.702513 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.702516 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.702517 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.702575 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.702802 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.702803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.702803 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.702942 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:51.703088 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:51.703180 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 05:41:51.703259 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":69,"threads_started":1}
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[27935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401311, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:51.710901 28019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:51.704597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60626 (local address 127.23.213.254:40421)
0926 05:41:51.704656 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:51.704658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:51.704679 (+ 21us) server_negotiation.cc:407] Connection header received
0926 05:41:51.704740 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:51.704741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:51.704750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:51.704796 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:51.705193 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.705661 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.706338 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:51.706513 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:51.706560 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:51.708601 (+ 2041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:51.708602 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:51.708603 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:51.708609 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:51.710000 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.710284 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.710285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.710285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.710333 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:51.710471 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:51.710471 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:51.710471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:51.710589 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:51.710666 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:51.710747 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 05:41:51.710802 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 05:41:51.711090 27980 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60626
I20260926 05:41:51.711387 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28042
I20260926 05:41:51.716041 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28176
I20260926 05:41:51.720245 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28310
I20260926 05:41:51.724067 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 27948
2026-09-26T05:41:51Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1868 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[28451](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[28451](info): set up 2 sockets
Sep 26 05:41:51 dist-test-slave-1fdt krb5kdc[28451](info): commencing operation
krb5kdc: starting...
W20260926 05:41:52.753783 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:52 dist-test-slave-1fdt krb5kdc[28451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:52.870541 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33063
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:33063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:52.951517 28464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:52.951749 28464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:52.951787 28464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:52.952948 28464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:52.953011 28464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:52.953035 28464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:52.953056 28464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:52.953089 28464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:52.954573 28464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:33063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28464
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:52.954838 28464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:52.955049 28464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:52.959717 28470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:52.959722 28464 server_base.cc:1034] running on GCE node
W20260926 05:41:52.959851 28472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:52.959823 28469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:52.960199 28464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:52.960438 28464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:52.961571 28464 hybrid_clock.cc:648] HybridClock initialized: now 1790401312961552 us; error 36 us; skew 500 ppm
Sep 26 05:41:52 dist-test-slave-1fdt krb5kdc[28451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401312, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:52.964036 28464 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:52.964370 28464 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:52.964419 28464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:52.964551 28464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:52.965564 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "5e4db11480dd44b5b5be4d6cb82d46dd"
format_stamp: "Formatted at 2026-09-26 05:41:52 on dist-test-slave-1fdt"
server_key: "d25798da91d1859ea67a8bd36711d5c5"
server_key_iv: "5048296f76b7e0a5de1fe5011caed89d"
server_key_version: "encryptionkey@0"
I20260926 05:41:52.965935 28464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "5e4db11480dd44b5b5be4d6cb82d46dd"
format_stamp: "Formatted at 2026-09-26 05:41:52 on dist-test-slave-1fdt"
server_key: "d25798da91d1859ea67a8bd36711d5c5"
server_key_iv: "5048296f76b7e0a5de1fe5011caed89d"
server_key_version: "encryptionkey@0"
I20260926 05:41:52.967196 28464 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:52.967906 28478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:52.968101 28464 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:52.968189 28464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "5e4db11480dd44b5b5be4d6cb82d46dd"
format_stamp: "Formatted at 2026-09-26 05:41:52 on dist-test-slave-1fdt"
server_key: "d25798da91d1859ea67a8bd36711d5c5"
server_key_iv: "5048296f76b7e0a5de1fe5011caed89d"
server_key_version: "encryptionkey@0"
I20260926 05:41:52.968245 28464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:52.976334 28464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:52.978163 28464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:52.978271 28464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:52.981990 28464 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:33063
I20260926 05:41:52.982014 28530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:33063 every 8 connection(s)
I20260926 05:41:52.982291 28464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:52.983207 28531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:52.984913 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28464
I20260926 05:41:52.985033 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:52.985209 24407 external_mini_cluster.cc:1428] Setting key f87db2f0bbfbafb48c50a1f94d3bffef
I20260926 05:41:52.985908 28531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd: Bootstrap starting.
I20260926 05:41:52.986500 28531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:52.986835 28531 log.cc:824] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:52.987564 28531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd: No bootstrap required, opened a new log
I20260926 05:41:52.989609 28531 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33063 } }
I20260926 05:41:52.989735 28531 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:52.989840 28531 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e4db11480dd44b5b5be4d6cb82d46dd, State: Initialized, Role: FOLLOWER
I20260926 05:41:52.989971 28531 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33063 } }
I20260926 05:41:52.990070 28531 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:52.990125 28531 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:52.990175 28531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:52.990773 28531 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33063 } }
I20260926 05:41:52.990886 28531 leader_election.cc:302] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5e4db11480dd44b5b5be4d6cb82d46dd; no voters:
I20260926 05:41:52.991065 28531 leader_election.cc:288] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:52.991104 28536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:52.991210 28536 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [term 1 LEADER]: Becoming Leader. State: Replica: 5e4db11480dd44b5b5be4d6cb82d46dd, State: Running, Role: LEADER
I20260926 05:41:52.991288 28536 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33063 } }
I20260926 05:41:52.991325 28531 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:52.991600 28536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e4db11480dd44b5b5be4d6cb82d46dd. Latest consensus state: current_term: 1 leader_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33063 } } }
I20260926 05:41:52.991679 28536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:52.991603 28538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5e4db11480dd44b5b5be4d6cb82d46dd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33063 } } }
I20260926 05:41:52.991859 28538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:52.991902 28541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 05:41:52 dist-test-slave-1fdt krb5kdc[28451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401312, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:52.992413 28541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:52.994571 28541 catalog_manager.cc:1347] Generated new cluster ID: 22762cc839bb4114919c558aadafabbb
I20260926 05:41:52.994630 28541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:52.996232 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:52.985721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54550 (local address 127.23.213.254:33063)
0926 05:41:52.985911 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:52.985924 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:52.985947 (+ 23us) server_negotiation.cc:407] Connection header received
0926 05:41:52.986417 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:52.986423 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:52.986468 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:52.986623 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:52.987379 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:52.988273 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:52.989363 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:52.989811 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:52.989910 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:52.992555 (+ 2645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:52.992559 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:52.992560 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:52.992570 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:52.994646 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:52.995021 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:52.995024 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:52.995024 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:52.995127 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:52.995323 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:52.995324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:52.995324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:52.995637 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:52.995771 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:52.995886 (+ 115us) server_negotiation.cc:299] Negotiation successful
0926 05:41:52.996027 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":110,"threads_started":1}
I20260926 05:41:53.010864 28541 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:53.011766 28541 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:53.014905 28541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd: Generated new TSK 0
I20260926 05:41:53.015110 28541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:53.044369 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33063
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:53.144362 28557 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:53.144564 28557 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:53.144589 28557 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:53.145792 28557 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:53.145829 28557 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:53.145856 28557 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:53.147297 28557 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28557
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:53.147485 28557 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:53.147657 28557 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:53.152580 28563 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.152604 28565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.152604 28562 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:53.152776 28557 server_base.cc:1034] running on GCE node
I20260926 05:41:53.152999 28557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:53.153227 28557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:53.154422 28557 hybrid_clock.cc:648] HybridClock initialized: now 1790401313154402 us; error 45 us; skew 500 ppm
Sep 26 05:41:53 dist-test-slave-1fdt krb5kdc[28451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:53.157006 28557 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:53.157370 28557 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:53.157454 28557 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:53.157606 28557 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:53.158840 28557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "3d667a71f2144362b383c0c255e8dda3"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "291313b7c215ac7ab3b6725316636077"
server_key_iv: "1d20f89933921927b6396bacd9f13fca"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.159220 28557 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "3d667a71f2144362b383c0c255e8dda3"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "291313b7c215ac7ab3b6725316636077"
server_key_iv: "1d20f89933921927b6396bacd9f13fca"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.160570 28557 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:53.161298 28571 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.161468 28557 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:53.161572 28557 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "3d667a71f2144362b383c0c255e8dda3"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "291313b7c215ac7ab3b6725316636077"
server_key_iv: "1d20f89933921927b6396bacd9f13fca"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.161656 28557 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:53.168818 28557 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:53.170671 28557 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:53.170877 28557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:53.171108 28557 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:53.171404 28557 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:53.171454 28557 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.171494 28557 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:53.171523 28557 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.177137 28557 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:45949
I20260926 05:41:53.177208 28684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:45949 every 8 connection(s)
I20260926 05:41:53.177464 28557 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:53.179584 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28557
I20260926 05:41:53.179677 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:53.179849 24407 external_mini_cluster.cc:1428] Setting key 0339399de83f8650999c58793c494a5d
Sep 26 05:41:53 dist-test-slave-1fdt krb5kdc[28451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:53.187225 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.178163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:53545 (local address 127.23.213.254:33063)
0926 05:41:53.178221 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.178224 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.178419 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:41:53.178870 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.178871 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.178881 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.178931 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:53.179910 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.180617 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.181636 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.181930 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.182008 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.184335 (+ 2327us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:53.184337 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:53.184337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:53.184345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:53.185876 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.186279 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.186280 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.186281 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.186340 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.186585 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.186586 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.186586 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.186700 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:53.186772 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.187043 (+ 271us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.187132 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
I20260926 05:41:53.187412 28687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.178161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:33063 (local address 127.23.213.193:53545)
0926 05:41:53.178308 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.178342 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.178790 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.179019 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.179022 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.179262 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:53.179818 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.179824 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.180703 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.180704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.181545 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.181548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.182024 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.182025 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.182159 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.182667 (+ 508us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:53.182675 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:53.184237 (+ 1562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:53.185969 (+ 1732us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:53.185977 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:53.185979 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:53.186193 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:53.186404 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:53.186405 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:53.186406 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:53.186516 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:53.186763 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:53.186770 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:53.186915 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.187023 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.187174 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":85,"threads_started":1}
I20260926 05:41:53.187868 28685 heartbeater.cc:347] Connected to a master server at 127.23.213.254:33063
I20260926 05:41:53.187968 28685 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:53.188127 28685 heartbeater.cc:511] Master 127.23.213.254:33063 requested a full tablet report, sending...
I20260926 05:41:53.188634 28495 ts_manager.cc:194] Registered new tserver with Master: 3d667a71f2144362b383c0c255e8dda3 (127.23.213.193:45949)
I20260926 05:41:53.189309 28495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:53545
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:53.202250 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33063
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:53.282322 28690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:53.282526 28690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:53.282549 28690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:53.283675 28690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:53.283708 28690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:53.283735 28690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:53.285348 28690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:53.285531 28690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:53.285706 28690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:53.290530 28698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.290585 28695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.290591 28696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:53.290678 28690 server_base.cc:1034] running on GCE node
I20260926 05:41:53.290928 28690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:53.291299 28690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:53.292502 28690 hybrid_clock.cc:648] HybridClock initialized: now 1790401313292494 us; error 40 us; skew 500 ppm
Sep 26 05:41:53 dist-test-slave-1fdt krb5kdc[28451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:53.294912 28690 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:53.295264 28690 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:53.295310 28690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:53.295398 28690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:53.296398 28690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "ea24a96823b041a1b07ef0b56a247411"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "edd54f5b2b1df4fce388afe7ca68cfce"
server_key_iv: "1cd0ef516581eb66c888a126ec776410"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.296698 28690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "ea24a96823b041a1b07ef0b56a247411"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "edd54f5b2b1df4fce388afe7ca68cfce"
server_key_iv: "1cd0ef516581eb66c888a126ec776410"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.297966 28690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:53.298647 28704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.298842 28690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:53.298913 28690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "ea24a96823b041a1b07ef0b56a247411"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "edd54f5b2b1df4fce388afe7ca68cfce"
server_key_iv: "1cd0ef516581eb66c888a126ec776410"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.298998 28690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:53.314708 28690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:53.316578 28690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:53.316725 28690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:53.316975 28690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:53.317294 28690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:53.317327 28690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.317366 28690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:53.317395 28690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.323177 28690 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:37945
I20260926 05:41:53.323215 28817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:37945 every 8 connection(s)
I20260926 05:41:53.323475 28690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:41:53.327026 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28690
I20260926 05:41:53.327147 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:53.327343 24407 external_mini_cluster.cc:1428] Setting key c7ff65710137ded6c9a285cde042e5e4
Sep 26 05:41:53 dist-test-slave-1fdt krb5kdc[28451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:53.333797 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.324176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:46001 (local address 127.23.213.254:33063)
0926 05:41:53.324225 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.324227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.324485 (+ 258us) server_negotiation.cc:407] Connection header received
0926 05:41:53.324947 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.324948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.324957 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.324994 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:53.325970 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.326455 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.327889 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.328172 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.328223 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.331021 (+ 2798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:53.331022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:53.331023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:53.331031 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:53.332400 (+ 1369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.332817 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.332818 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.332819 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.332877 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.333123 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.333125 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.333125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.333239 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:53.333313 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.333582 (+ 269us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.333676 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:41:53.333957 28820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.324194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:33063 (local address 127.23.213.194:46001)
0926 05:41:53.324351 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.324398 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.324860 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.325058 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.325061 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.325311 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:53.325885 (+ 574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.325892 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.326557 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.326558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.327796 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.327799 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.328257 (+ 458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.328259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.328439 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.328958 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:53.328969 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:53.330933 (+ 1964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:53.332500 (+ 1567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:53.332507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:53.332509 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:53.332735 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:53.332939 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:53.332940 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:53.332940 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:53.333053 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:53.333302 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:53.333308 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:53.333451 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.333551 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.333705 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
I20260926 05:41:53.334431 28818 heartbeater.cc:347] Connected to a master server at 127.23.213.254:33063
I20260926 05:41:53.334522 28818 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:53.334687 28818 heartbeater.cc:511] Master 127.23.213.254:33063 requested a full tablet report, sending...
I20260926 05:41:53.335072 28495 ts_manager.cc:194] Registered new tserver with Master: ea24a96823b041a1b07ef0b56a247411 (127.23.213.194:37945)
I20260926 05:41:53.335448 28495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:46001
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:53.352454 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33063
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37505
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:53.433480 28823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:53.433697 28823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:53.433722 28823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:53.434903 28823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:53.434952 28823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:53.434980 28823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:53.436394 28823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37505
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:33063
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:53.436571 28823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:53.436738 28823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:53.441520 28831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.441520 28828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.441531 28829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:53.441701 28823 server_base.cc:1034] running on GCE node
I20260926 05:41:53.442026 28823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:53.442256 28823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:53.443415 28823 hybrid_clock.cc:648] HybridClock initialized: now 1790401313443397 us; error 28 us; skew 500 ppm
Sep 26 05:41:53 dist-test-slave-1fdt krb5kdc[28451](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:53.445979 28823 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:53.446312 28823 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:53.446379 28823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:53.446527 28823 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:53.447553 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "23696a0509ff423490e4a8713fefa92a"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "410b72b3322928027c85f47dcdf4dec5"
server_key_iv: "01efb3559ce2f7b034a3d81149259a68"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.447903 28823 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "23696a0509ff423490e4a8713fefa92a"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "410b72b3322928027c85f47dcdf4dec5"
server_key_iv: "01efb3559ce2f7b034a3d81149259a68"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.449151 28823 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:41:53.450009 28837 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.450188 28823 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:53.450304 28823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "23696a0509ff423490e4a8713fefa92a"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "410b72b3322928027c85f47dcdf4dec5"
server_key_iv: "01efb3559ce2f7b034a3d81149259a68"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.450398 28823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:53.495555 28823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:53.498114 28823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:53.498279 28823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:53.498531 28823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:53.498855 28823 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:53.498942 28823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.498991 28823 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:53.499023 28823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.505398 28823 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:43613
I20260926 05:41:53.505445 28950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:43613 every 8 connection(s)
I20260926 05:41:53.505726 28823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:53.507298 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28823
I20260926 05:41:53.507395 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:53.507571 24407 external_mini_cluster.cc:1428] Setting key 6b2158991803022856afde57e7def4ef
Sep 26 05:41:53 dist-test-slave-1fdt krb5kdc[28451](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401313, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:53.515107 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.506459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:34247 (local address 127.23.213.254:33063)
0926 05:41:53.506503 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.506505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.506710 (+ 205us) server_negotiation.cc:407] Connection header received
0926 05:41:53.507275 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.507277 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.507289 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.507351 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:53.508636 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.509173 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.510220 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.510413 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.510482 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.512492 (+ 2010us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:53.512493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:53.512494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:53.512500 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:53.513978 (+ 1478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.514309 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.514310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.514310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.514361 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.514550 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.514551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.514551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.514648 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:53.514715 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.514938 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.515035 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:41:53.515332 28953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.506431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:33063 (local address 127.23.213.195:34247)
0926 05:41:53.506587 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.506626 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.507148 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.507438 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.507443 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.507804 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:53.508527 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.508533 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.509253 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.509254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.510147 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.510149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.510496 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.510496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.510590 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.511026 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:53.511034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:53.512412 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:53.514042 (+ 1630us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:53.514046 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:53.514047 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:53.514249 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:53.514411 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:53.514411 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:53.514412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:53.514503 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:53.514704 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:53.514708 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:53.514819 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.514899 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.515079 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":79,"threads_started":1}
I20260926 05:41:53.515769 28951 heartbeater.cc:347] Connected to a master server at 127.23.213.254:33063
I20260926 05:41:53.515865 28951 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:53.516016 28951 heartbeater.cc:511] Master 127.23.213.254:33063 requested a full tablet report, sending...
I20260926 05:41:53.516417 28495 ts_manager.cc:194] Registered new tserver with Master: 23696a0509ff423490e4a8713fefa92a (127.23.213.195:43613)
I20260926 05:41:53.516795 28495 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:34247
I20260926 05:41:53.519308 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:41:53.526289 28534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.520462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54562 (local address 127.23.213.254:33063)
0926 05:41:53.520519 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.520520 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.520650 (+ 130us) server_negotiation.cc:407] Connection header received
0926 05:41:53.520732 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.520733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.520743 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.520798 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:53.521327 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.521873 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.522583 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.522787 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.522866 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.523952 (+ 1086us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:53.523953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:53.523954 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:53.523961 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:53.525148 (+ 1187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.525481 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.525482 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.525483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.525535 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:53.525731 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:53.525732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:53.525732 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:53.525905 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:53.526010 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.526118 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.526191 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:41:53.527793 28495 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54562:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:41:53.529425 28495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:53.536182 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.531867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46792 (local address 127.23.213.193:45949)
0926 05:41:53.532064 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.532068 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.532080 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:41:53.532322 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.532324 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.532342 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.532410 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:53.532692 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.533846 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.534980 (+ 1134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.535719 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.535810 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.535870 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.536056 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.536097 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260926 05:41:53.536208 28962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.531764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:45949 (local address 127.0.0.1:46792)
0926 05:41:53.531918 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.531972 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.532229 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.532525 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.532525 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.532527 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:53.532606 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.532609 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.533974 (+ 1365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.533975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.534889 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.534891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.535826 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.535827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.535936 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.535952 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.536033 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.536104 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":26,"thread_start_us":51,"threads_started":1}
I20260926 05:41:53.536504 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.532083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42018 (local address 127.23.213.194:37945)
0926 05:41:53.532242 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.532247 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.532257 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:41:53.532297 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.532299 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.532315 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.532415 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:53.532674 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.533822 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.535076 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.535741 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.535818 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.535870 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.536350 (+ 480us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.536419 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":97,"threads_started":1}
I20260926 05:41:53.536209 28963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.531831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:37945 (local address 127.0.0.1:42018)
0926 05:41:53.532126 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.532136 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.532188 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.532512 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.532516 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.532519 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:53.532588 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.532590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.533918 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.533919 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.534999 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.535000 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.535824 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.535826 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.535971 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.535974 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.536033 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.536104 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":276,"mutex_wait_us":31,"thread_start_us":96,"threads_started":1}
I20260926 05:41:53.537127 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.531831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:43613 (local address 127.0.0.1:49406)
0926 05:41:53.532295 (+ 464us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.532304 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.532489 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.532752 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.532753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.532754 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:53.532794 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.532795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.533669 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.533670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.534993 (+ 1323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.534995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.536842 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.536843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.536911 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.536915 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.537006 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.537056 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":446,"mutex_wait_us":14,"thread_start_us":89,"threads_started":1}
I20260926 05:41:53.537140 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.532092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49406 (local address 127.23.213.195:43613)
0926 05:41:53.532358 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.532362 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.532546 (+ 184us) server_negotiation.cc:407] Connection header received
0926 05:41:53.532590 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.532592 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.532607 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.532677 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:53.532863 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.533568 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.535909 (+ 2341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.536686 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.536753 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.536807 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.537025 (+ 218us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.537067 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":235,"thread_start_us":85,"threads_started":1}
I20260926 05:41:53.537298 28619 tablet_service.cc:1467] Processing CreateTablet for tablet bff11a60c19244d485b52a4481b86767 (DEFAULT_TABLE table=test-table [id=0a556906e59e4cda94cd3b6c020a9893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:53.537528 28752 tablet_service.cc:1467] Processing CreateTablet for tablet bff11a60c19244d485b52a4481b86767 (DEFAULT_TABLE table=test-table [id=0a556906e59e4cda94cd3b6c020a9893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:53.537679 28619 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff11a60c19244d485b52a4481b86767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:53.537819 28752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff11a60c19244d485b52a4481b86767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:53.537773 28885 tablet_service.cc:1467] Processing CreateTablet for tablet bff11a60c19244d485b52a4481b86767 (DEFAULT_TABLE table=test-table [id=0a556906e59e4cda94cd3b6c020a9893]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:53.538066 28885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bff11a60c19244d485b52a4481b86767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:53.540230 28968 tablet_bootstrap.cc:492] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Bootstrap starting.
I20260926 05:41:53.540815 28968 tablet_bootstrap.cc:654] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:53.541201 28969 tablet_bootstrap.cc:492] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Bootstrap starting.
I20260926 05:41:53.541421 28968 log.cc:824] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:53.541822 28970 tablet_bootstrap.cc:492] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: Bootstrap starting.
I20260926 05:41:53.541957 28969 tablet_bootstrap.cc:654] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:53.542387 28970 tablet_bootstrap.cc:654] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:53.542418 28969 log.cc:824] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:53.542879 28970 log.cc:824] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:53.543104 28969 tablet_bootstrap.cc:492] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: No bootstrap required, opened a new log
I20260926 05:41:53.543179 28969 ts_tablet_manager.cc:1397] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:53.543535 28970 tablet_bootstrap.cc:492] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: No bootstrap required, opened a new log
I20260926 05:41:53.543587 28970 ts_tablet_manager.cc:1397] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:53.544845 28969 raft_consensus.cc:348] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.545001 28969 raft_consensus.cc:374] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:53.545043 28969 raft_consensus.cc:729] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23696a0509ff423490e4a8713fefa92a, State: Initialized, Role: FOLLOWER
I20260926 05:41:53.545073 28970 raft_consensus.cc:348] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.545179 28969 consensus_queue.cc:260] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.545239 28970 raft_consensus.cc:374] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:53.545297 28970 raft_consensus.cc:729] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3d667a71f2144362b383c0c255e8dda3, State: Initialized, Role: FOLLOWER
I20260926 05:41:53.545404 28970 consensus_queue.cc:260] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.545470 28969 ts_tablet_manager.cc:1428] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:41:53.545534 28951 heartbeater.cc:503] Master 127.23.213.254:33063 was elected leader, sending a full tablet report...
I20260926 05:41:53.545686 28970 ts_tablet_manager.cc:1428] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:53.545706 28685 heartbeater.cc:503] Master 127.23.213.254:33063 was elected leader, sending a full tablet report...
I20260926 05:41:53.546147 28968 tablet_bootstrap.cc:492] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: No bootstrap required, opened a new log
I20260926 05:41:53.546222 28968 ts_tablet_manager.cc:1397] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 05:41:53.547811 28968 raft_consensus.cc:348] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.547971 28968 raft_consensus.cc:374] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:53.548009 28968 raft_consensus.cc:729] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea24a96823b041a1b07ef0b56a247411, State: Initialized, Role: FOLLOWER
I20260926 05:41:53.548115 28968 consensus_queue.cc:260] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.548374 28968 ts_tablet_manager.cc:1428] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:41:53.548395 28818 heartbeater.cc:503] Master 127.23.213.254:33063 was elected leader, sending a full tablet report...
W20260926 05:41:53.573953 28819 tablet.cc:2369] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:41:53.678143 28686 tablet.cc:2369] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:41:53.756295 28952 tablet.cc:2369] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:41:53.803367 28975 raft_consensus.cc:482] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:53.803519 28975 raft_consensus.cc:504] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.803918 28975 leader_election.cc:288] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23696a0509ff423490e4a8713fefa92a (127.23.213.195:43613), ea24a96823b041a1b07ef0b56a247411 (127.23.213.194:37945)
I20260926 05:41:53.807945 28966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.804167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:40375 (local address 127.23.213.194:37945)
0926 05:41:53.804221 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.804223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.804302 (+ 79us) server_negotiation.cc:407] Connection header received
0926 05:41:53.804347 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.804348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.804359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.804447 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:53.804641 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.805410 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.806997 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.807584 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.807666 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.807675 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.807833 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.807875 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:41:53.807986 28967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.804204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:42799 (local address 127.23.213.195:43613)
0926 05:41:53.804250 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.804252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.804305 (+ 53us) server_negotiation.cc:407] Connection header received
0926 05:41:53.804347 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.804348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.804359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.804447 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:41:53.804648 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.805401 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.806997 (+ 1596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.807581 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.807666 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.807675 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.807833 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.807875 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:41:53.807991 28978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.804083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:43613 (local address 127.23.213.193:42799)
0926 05:41:53.804216 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.804228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.804293 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.804464 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.804465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.804467 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:53.804565 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.804566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.805500 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.805500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.806883 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.806886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.807679 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.807680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.807753 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.807755 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.807814 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.807875 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":25,"thread_start_us":61,"threads_started":1}
I20260926 05:41:53.807991 28977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.804084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:37945 (local address 127.23.213.193:40375)
0926 05:41:53.804216 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.804228 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:53.804293 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:53.804464 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:53.804465 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:53.804468 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:41:53.804545 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.804548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.805500 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.805500 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.806883 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:53.806886 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.807678 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.807679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:53.807750 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.807754 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:41:53.807814 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:41:53.807875 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":94,"thread_start_us":66,"threads_started":1}
I20260926 05:41:53.808416 28905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff11a60c19244d485b52a4481b86767" candidate_uuid: "3d667a71f2144362b383c0c255e8dda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23696a0509ff423490e4a8713fefa92a" is_pre_election: true
I20260926 05:41:53.808509 28772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff11a60c19244d485b52a4481b86767" candidate_uuid: "3d667a71f2144362b383c0c255e8dda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea24a96823b041a1b07ef0b56a247411" is_pre_election: true
I20260926 05:41:53.808589 28905 raft_consensus.cc:2454] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d667a71f2144362b383c0c255e8dda3 in term 0.
I20260926 05:41:53.808650 28772 raft_consensus.cc:2454] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3d667a71f2144362b383c0c255e8dda3 in term 0.
I20260926 05:41:53.808851 28573 leader_election.cc:302] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23696a0509ff423490e4a8713fefa92a, 3d667a71f2144362b383c0c255e8dda3; no voters:
I20260926 05:41:53.809008 28975 raft_consensus.cc:2781] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:53.809108 28975 raft_consensus.cc:482] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:53.809142 28975 raft_consensus.cc:3037] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:53.809851 28975 raft_consensus.cc:504] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.810026 28975 leader_election.cc:288] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [CANDIDATE]: Term 1 election: Requested vote from peers 23696a0509ff423490e4a8713fefa92a (127.23.213.195:43613), ea24a96823b041a1b07ef0b56a247411 (127.23.213.194:37945)
I20260926 05:41:53.810160 28905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff11a60c19244d485b52a4481b86767" candidate_uuid: "3d667a71f2144362b383c0c255e8dda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23696a0509ff423490e4a8713fefa92a"
I20260926 05:41:53.810177 28772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bff11a60c19244d485b52a4481b86767" candidate_uuid: "3d667a71f2144362b383c0c255e8dda3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ea24a96823b041a1b07ef0b56a247411"
I20260926 05:41:53.810238 28905 raft_consensus.cc:3037] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:53.810259 28772 raft_consensus.cc:3037] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:53.810931 28905 raft_consensus.cc:2454] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d667a71f2144362b383c0c255e8dda3 in term 1.
I20260926 05:41:53.810933 28772 raft_consensus.cc:2454] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3d667a71f2144362b383c0c255e8dda3 in term 1.
I20260926 05:41:53.811111 28573 leader_election.cc:302] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23696a0509ff423490e4a8713fefa92a, 3d667a71f2144362b383c0c255e8dda3; no voters:
I20260926 05:41:53.811213 28975 raft_consensus.cc:2781] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:53.811363 28975 raft_consensus.cc:686] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 1 LEADER]: Becoming Leader. State: Replica: 3d667a71f2144362b383c0c255e8dda3, State: Running, Role: LEADER
I20260926 05:41:53.811460 28975 consensus_queue.cc:237] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } }
I20260926 05:41:53.812359 28495 catalog_manager.cc:5495] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3d667a71f2144362b383c0c255e8dda3 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "3d667a71f2144362b383c0c255e8dda3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3d667a71f2144362b383c0c255e8dda3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45949 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:53.818495 28965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.815893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46808 (local address 127.23.213.193:45949)
0926 05:41:53.815976 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.815978 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.815985 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:41:53.816017 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.816018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.816033 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.816109 (+ 76us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:41:53.816255 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.816720 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.817596 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:53.817845 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:53.817970 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:53.818180 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:41:53.818287 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:41:53.818380 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.818398 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.818421 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 05:41:53.820329 28772 raft_consensus.cc:1261] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 1 FOLLOWER]: Refusing update from remote peer 3d667a71f2144362b383c0c255e8dda3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:53.820338 28905 raft_consensus.cc:1261] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 1 FOLLOWER]: Refusing update from remote peer 3d667a71f2144362b383c0c255e8dda3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:53.820662 28975 consensus_queue.cc:1035] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ea24a96823b041a1b07ef0b56a247411" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:53.820806 28979 consensus_queue.cc:1035] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23696a0509ff423490e4a8713fefa92a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 43613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:53.823225 28983 mvcc.cc:205] Tried to move back new op lower bound from 7333483781405478912 to 7333483781372370944. Current Snapshot: MvccSnapshot[applied={T|T < 7333483781405478912}]
I20260926 05:41:53.824556 28984 mvcc.cc:205] Tried to move back new op lower bound from 7333483781405478912 to 7333483781372370944. Current Snapshot: MvccSnapshot[applied={T|T < 7333483781405478912 or (T in {7333483781405478912})}]
I20260926 05:41:53.825663 28495 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:53.825784 28495 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54562:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:53.826560 28495 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5e4db11480dd44b5b5be4d6cb82d46dd: Sending DeleteTablet for 3 replicas of tablet bff11a60c19244d485b52a4481b86767
I20260926 05:41:53.826859 28619 tablet_service.cc:1514] Processing DeleteTablet for tablet bff11a60c19244d485b52a4481b86767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:53 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:46792
I20260926 05:41:53.826941 28885 tablet_service.cc:1514] Processing DeleteTablet for tablet bff11a60c19244d485b52a4481b86767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:53 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:49406
I20260926 05:41:53.826967 28752 tablet_service.cc:1514] Processing DeleteTablet for tablet bff11a60c19244d485b52a4481b86767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:53 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42018
I20260926 05:41:53.827135 28991 tablet_replica.cc:330] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3: stopping tablet replica
I20260926 05:41:53.827219 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28557
I20260926 05:41:53.827245 28991 raft_consensus.cc:2229] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:41:53.827343 28991 raft_consensus.cc:2258] T bff11a60c19244d485b52a4481b86767 P 3d667a71f2144362b383c0c255e8dda3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:53.827472 28992 tablet_replica.cc:330] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: stopping tablet replica
I20260926 05:41:53.827544 28992 raft_consensus.cc:2229] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:53.827721 28992 raft_consensus.cc:2258] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:53.828203 28992 ts_tablet_manager.cc:1905] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:53.829839 28993 tablet_replica.cc:330] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: stopping tablet replica
I20260926 05:41:53.829947 28993 raft_consensus.cc:2229] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:53.830030 28992 ts_tablet_manager.cc:1918] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:41:53.830055 28993 raft_consensus.cc:2258] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:53.830111 28992 log.cc:1196] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/bff11a60c19244d485b52a4481b86767
I20260926 05:41:53.830399 28992 ts_tablet_manager.cc:1939] T bff11a60c19244d485b52a4481b86767 P 23696a0509ff423490e4a8713fefa92a: Deleting consensus metadata
I20260926 05:41:53.830664 28993 ts_tablet_manager.cc:1905] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:53.830905 28480 catalog_manager.cc:4841] TS 23696a0509ff423490e4a8713fefa92a (127.23.213.195:43613): tablet bff11a60c19244d485b52a4481b86767 (table test-table [id=0a556906e59e4cda94cd3b6c020a9893]) successfully deleted
I20260926 05:41:53.832109 28993 ts_tablet_manager.cc:1918] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:41:53.832168 28993 log.cc:1196] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/bff11a60c19244d485b52a4481b86767
I20260926 05:41:53.832425 28993 ts_tablet_manager.cc:1939] T bff11a60c19244d485b52a4481b86767 P ea24a96823b041a1b07ef0b56a247411: Deleting consensus metadata
I20260926 05:41:53.832824 28482 catalog_manager.cc:4841] TS ea24a96823b041a1b07ef0b56a247411 (127.23.213.194:37945): tablet bff11a60c19244d485b52a4481b86767 (table test-table [id=0a556906e59e4cda94cd3b6c020a9893]) successfully deleted
W20260926 05:41:53.833665 28481 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:45949 (error 108)
I20260926 05:41:53.833811 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28690
I20260926 05:41:53.834692 28964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.834504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:45949 (local address 127.0.0.1:46814)
0926 05:41:53.834605 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:53.834651 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:45949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 05:41:53.834805 28481 catalog_manager.cc:4584] TS 3d667a71f2144362b383c0c255e8dda3 (127.23.213.193:45949): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bff11a60c19244d485b52a4481b86767: Network error: Client connection negotiation failed: client connection to 127.23.213.193:45949: connect: Connection refused (error 111)
I20260926 05:41:53.839486 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28823
I20260926 05:41:53.844512 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28464
2026-09-26T05:41:53Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (2120 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T05:41:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:53Z Disabled control of system clock
I20260926 05:41:53.871697 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38315
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:37279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38315
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:41:53.951877 28998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:53.952090 28998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:53.952112 28998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:53.953354 28998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:53.953397 28998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:53.953410 28998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:53.953423 28998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:53.953435 28998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:53.955338 28998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38315
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38315
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:53.955662 28998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:53.956008 28998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:53.961392 29004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.961385 29003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:53.961385 29006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:53.961671 28998 server_base.cc:1034] running on GCE node
I20260926 05:41:53.962061 28998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:53.962316 28998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:53.963454 28998 hybrid_clock.cc:648] HybridClock initialized: now 1790401313963438 us; error 36 us; skew 500 ppm
I20260926 05:41:53.964702 28998 webserver.cc:460] Webserver started at http://127.23.213.254:40495/ using document root <none> and password file <none>
I20260926 05:41:53.964950 28998 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:53.965018 28998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:53.965180 28998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:53.966333 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "81a12daec7e14ab792da5502010f29ea"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "771b0ea88866677091f2ce318393810b"
server_key_iv: "35b3eccbd7967b728a2d6ffe73764e1e"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.966692 28998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "81a12daec7e14ab792da5502010f29ea"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "771b0ea88866677091f2ce318393810b"
server_key_iv: "35b3eccbd7967b728a2d6ffe73764e1e"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.968048 28998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:53.968797 29012 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:53.968994 28998 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:53.969106 28998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "81a12daec7e14ab792da5502010f29ea"
format_stamp: "Formatted at 2026-09-26 05:41:53 on dist-test-slave-1fdt"
server_key: "771b0ea88866677091f2ce318393810b"
server_key_iv: "35b3eccbd7967b728a2d6ffe73764e1e"
server_key_version: "encryptionkey@0"
I20260926 05:41:53.969183 28998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:53.972561 28998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:53.972827 28998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:53.972966 28998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:53.976984 28998 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38315
I20260926 05:41:53.977023 29064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38315 every 8 connection(s)
I20260926 05:41:53.977358 28998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:53.978093 29065 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:53.980877 29065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea: Bootstrap starting.
I20260926 05:41:53.981418 29065 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:53.981662 29065 log.cc:824] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:53.982436 29065 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea: No bootstrap required, opened a new log
I20260926 05:41:53.984022 29065 raft_consensus.cc:348] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a12daec7e14ab792da5502010f29ea" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38315 } }
I20260926 05:41:53.984156 29065 raft_consensus.cc:374] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:53.984179 29065 raft_consensus.cc:729] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 81a12daec7e14ab792da5502010f29ea, State: Initialized, Role: FOLLOWER
I20260926 05:41:53.984318 29065 consensus_queue.cc:260] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a12daec7e14ab792da5502010f29ea" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38315 } }
I20260926 05:41:53.984409 29065 raft_consensus.cc:388] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:53.984433 29065 raft_consensus.cc:482] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:53.984465 29065 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:53.984956 29065 raft_consensus.cc:504] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a12daec7e14ab792da5502010f29ea" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38315 } }
I20260926 05:41:53.985037 29065 leader_election.cc:302] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 81a12daec7e14ab792da5502010f29ea; no voters:
I20260926 05:41:53.985155 29065 leader_election.cc:288] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:53.985219 29068 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:53.985389 29068 raft_consensus.cc:686] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [term 1 LEADER]: Becoming Leader. State: Replica: 81a12daec7e14ab792da5502010f29ea, State: Running, Role: LEADER
I20260926 05:41:53.985388 29065 sys_catalog.cc:564] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:53.985529 29068 consensus_queue.cc:237] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a12daec7e14ab792da5502010f29ea" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38315 } }
I20260926 05:41:53.985947 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 28998
I20260926 05:41:53.986052 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:53.986145 29070 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [sys.catalog]: SysCatalogTable state changed. Reason: New leader 81a12daec7e14ab792da5502010f29ea. Latest consensus state: current_term: 1 leader_uuid: "81a12daec7e14ab792da5502010f29ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a12daec7e14ab792da5502010f29ea" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38315 } } }
I20260926 05:41:53.986228 29070 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:53.986263 24407 external_mini_cluster.cc:1428] Setting key 5d312482a24c4d5abbd8e41ba9b9ab21
I20260926 05:41:53.986512 29069 sys_catalog.cc:455] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "81a12daec7e14ab792da5502010f29ea" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "81a12daec7e14ab792da5502010f29ea" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38315 } } }
I20260926 05:41:53.986568 29069 sys_catalog.cc:458] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:53.986649 29077 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:53.987131 29077 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:53.988782 29077 catalog_manager.cc:1347] Generated new cluster ID: 94f05731064648a7b61ca673e6d0b09c
I20260926 05:41:53.988850 29077 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:53.988924 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:53.986913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38298 (local address 127.23.213.254:38315)
0926 05:41:53.987265 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:53.987275 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:53.987309 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:41:53.987420 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:53.987427 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:53.987477 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:53.987610 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:53.988423 (+ 813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:53.988427 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:53.988427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:53.988432 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:53.988480 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:53.988494 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:53.988560 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:53.988643 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 05:41:53.988700 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260926 05:41:54.000696 29077 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:54.000911 29077 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:54.013676 29077 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea: Generated new TSK 0
I20260926 05:41:54.013973 29077 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:41:54.019598 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38315
--builtin_ntp_servers=127.23.213.212:37279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:41:54.101483 29089 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:54.101689 29089 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:54.101711 29089 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:54.102914 29089 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:54.102955 29089 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:54.102981 29089 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:54.104403 29089 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29089
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:54.104655 29089 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:54.104960 29089 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:54.111229 29095 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:54.111337 29089 server_base.cc:1034] running on GCE node
W20260926 05:41:54.111229 29094 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:54.111505 29097 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:54.111733 29089 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:54.112015 29089 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:54.113193 29089 hybrid_clock.cc:648] HybridClock initialized: now 1790401314113175 us; error 36 us; skew 500 ppm
I20260926 05:41:54.114446 29089 webserver.cc:460] Webserver started at http://127.23.213.193:38961/ using document root <none> and password file <none>
I20260926 05:41:54.114701 29089 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:54.114748 29089 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:54.114890 29089 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:54.115998 29089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "5da19530eeab43e6855a416c269a9610"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "25e6c0910e44039d6a91565548f0a81f"
server_key_iv: "74055db4201a8516d535970bf204f9c2"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.116366 29089 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "5da19530eeab43e6855a416c269a9610"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "25e6c0910e44039d6a91565548f0a81f"
server_key_iv: "74055db4201a8516d535970bf204f9c2"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.117679 29089 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:54.118559 29103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.118774 29089 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:54.118880 29089 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "5da19530eeab43e6855a416c269a9610"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "25e6c0910e44039d6a91565548f0a81f"
server_key_iv: "74055db4201a8516d535970bf204f9c2"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.118957 29089 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:54.122355 29089 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:54.122622 29089 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:54.122761 29089 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:54.122965 29089 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:54.123266 29089 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:54.123318 29089 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.123358 29089 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:54.123385 29089 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.129277 29089 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:40231
I20260926 05:41:54.129321 29216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:40231 every 8 connection(s)
I20260926 05:41:54.129642 29089 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:54.132530 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.130421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:54957 (local address 127.23.213.254:38315)
0926 05:41:54.130507 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.130509 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.130708 (+ 199us) server_negotiation.cc:407] Connection header received
0926 05:41:54.131171 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.131172 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.131179 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.131246 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.132331 (+ 1085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.132332 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.132332 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.132337 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.132352 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.132356 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.132417 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.132464 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.132477 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:41:54.132726 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.130407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38315 (local address 127.23.213.193:54957)
0926 05:41:54.130580 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.130619 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.131088 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.131300 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.131304 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.131311 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.132171 (+ 860us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.132172 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.132198 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.132224 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.132416 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.132418 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.132421 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.132470 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.132515 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260926 05:41:54.133430 29217 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38315
I20260926 05:41:54.133530 29217 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:54.133693 29217 heartbeater.cc:511] Master 127.23.213.254:38315 requested a full tablet report, sending...
I20260926 05:41:54.134296 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29089
I20260926 05:41:54.134308 29029 ts_manager.cc:194] Registered new tserver with Master: 5da19530eeab43e6855a416c269a9610 (127.23.213.193:40231)
I20260926 05:41:54.134366 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:54.134654 24407 external_mini_cluster.cc:1428] Setting key 0fcceabb246e29b740bb7c7f62da8235
I20260926 05:41:54.135430 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38315
--builtin_ntp_servers=127.23.213.212:37279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:41:54.219506 29220 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:54.219702 29220 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:54.219723 29220 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:54.220824 29220 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:54.220857 29220 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:54.220881 29220 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:54.222299 29220 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29220
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:54.222538 29220 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:54.222822 29220 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:54.227756 29228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:54.227785 29226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:54.227777 29225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:54.227993 29220 server_base.cc:1034] running on GCE node
I20260926 05:41:54.228206 29220 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:54.228400 29220 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:54.229534 29220 hybrid_clock.cc:648] HybridClock initialized: now 1790401314229510 us; error 37 us; skew 500 ppm
I20260926 05:41:54.230813 29220 webserver.cc:460] Webserver started at http://127.23.213.194:43197/ using document root <none> and password file <none>
I20260926 05:41:54.231063 29220 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:54.231132 29220 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:54.231295 29220 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:54.232326 29220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "2fa4d408e9d9439695def5dca99a5797"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "5a594c59e0254240ee9ab8da41cd86c5"
server_key_iv: "84396d7ccaacb04212b2bf70ed1e0abb"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.232666 29220 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "2fa4d408e9d9439695def5dca99a5797"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "5a594c59e0254240ee9ab8da41cd86c5"
server_key_iv: "84396d7ccaacb04212b2bf70ed1e0abb"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.233958 29220 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:54.234654 29234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.234822 29220 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.234921 29220 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "2fa4d408e9d9439695def5dca99a5797"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "5a594c59e0254240ee9ab8da41cd86c5"
server_key_iv: "84396d7ccaacb04212b2bf70ed1e0abb"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.234997 29220 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:54.238381 29220 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:54.238638 29220 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:54.238770 29220 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:54.238981 29220 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:54.239269 29220 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:54.239320 29220 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.239357 29220 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:54.239384 29220 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.245004 29220 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:36019
I20260926 05:41:54.245046 29347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:36019 every 8 connection(s)
I20260926 05:41:54.245362 29220 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:41:54.248224 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.246039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:51739 (local address 127.23.213.254:38315)
0926 05:41:54.246125 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.246127 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.246331 (+ 204us) server_negotiation.cc:407] Connection header received
0926 05:41:54.246791 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.246792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.246797 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.246849 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.248021 (+ 1172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.248022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.248023 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.248026 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.248039 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.248042 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.248097 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.248157 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.248169 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:41:54.248440 29350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.246006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38315 (local address 127.23.213.194:51739)
0926 05:41:54.246215 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.246247 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.246705 (+ 458us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.246902 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.246905 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.246913 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.247878 (+ 965us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.247878 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.247907 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.247938 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.248096 (+ 158us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.248098 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.248101 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.248153 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.248199 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":115,"threads_started":1}
I20260926 05:41:54.248903 29348 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38315
I20260926 05:41:54.249011 29348 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:54.249161 29348 heartbeater.cc:511] Master 127.23.213.254:38315 requested a full tablet report, sending...
I20260926 05:41:54.249542 29029 ts_manager.cc:194] Registered new tserver with Master: 2fa4d408e9d9439695def5dca99a5797 (127.23.213.194:36019)
I20260926 05:41:54.249991 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29220
I20260926 05:41:54.250093 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:54.250264 24407 external_mini_cluster.cc:1428] Setting key 70736673ca0f686ac4b092f06be7acef
I20260926 05:41:54.251118 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38315
--builtin_ntp_servers=127.23.213.212:37279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 05:41:54.332331 29351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:54.332554 29351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:54.332576 29351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:54.333807 29351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:54.333846 29351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:54.333873 29351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:54.335202 29351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38315
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:54.335477 29351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:54.335784 29351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:54.340816 29359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:54.340844 29357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:54.340974 29351 server_base.cc:1034] running on GCE node
W20260926 05:41:54.340816 29356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:54.341295 29351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:54.341563 29351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:54.342736 29351 hybrid_clock.cc:648] HybridClock initialized: now 1790401314342697 us; error 54 us; skew 500 ppm
I20260926 05:41:54.344090 29351 webserver.cc:460] Webserver started at http://127.23.213.195:39323/ using document root <none> and password file <none>
I20260926 05:41:54.344376 29351 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:54.344448 29351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:54.344586 29351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:54.345654 29351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "d7cb6f5748b8457a83829568ef3080c0"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "1a958d993b9a1637c875bcb138eea8ce"
server_key_iv: "2840e3f5c25f28858971a61b8a315ddf"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.346086 29351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "d7cb6f5748b8457a83829568ef3080c0"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "1a958d993b9a1637c875bcb138eea8ce"
server_key_iv: "2840e3f5c25f28858971a61b8a315ddf"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.347357 29351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:54.348075 29365 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.348253 29351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:54.348348 29351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "d7cb6f5748b8457a83829568ef3080c0"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "1a958d993b9a1637c875bcb138eea8ce"
server_key_iv: "2840e3f5c25f28858971a61b8a315ddf"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.348428 29351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:54.351828 29351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:54.352087 29351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:54.352217 29351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:54.352411 29351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:54.352705 29351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:54.352754 29351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.352793 29351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:54.352820 29351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.358538 29351 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:37381
I20260926 05:41:54.358600 29478 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:37381 every 8 connection(s)
I20260926 05:41:54.358916 29351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:54.361725 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.359685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:59713 (local address 127.23.213.254:38315)
0926 05:41:54.359746 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.359748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.359944 (+ 196us) server_negotiation.cc:407] Connection header received
0926 05:41:54.360382 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.360383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.360389 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.360451 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.361514 (+ 1063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.361516 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.361516 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.361521 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.361535 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.361539 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.361597 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.361661 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.361674 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:41:54.361979 29481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.359666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38315 (local address 127.23.213.195:59713)
0926 05:41:54.359839 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.359871 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.360301 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.360500 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.360503 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.360510 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.361348 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.361348 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.361375 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.361401 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.361604 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.361607 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.361609 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.361660 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.361704 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":99,"threads_started":1}
I20260926 05:41:54.362445 29479 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38315
I20260926 05:41:54.362537 29479 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:54.362643 29479 heartbeater.cc:511] Master 127.23.213.254:38315 requested a full tablet report, sending...
I20260926 05:41:54.363031 29029 ts_manager.cc:194] Registered new tserver with Master: d7cb6f5748b8457a83829568ef3080c0 (127.23.213.195:37381)
I20260926 05:41:54.365656 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29351
I20260926 05:41:54.365777 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:54.365932 24407 external_mini_cluster.cc:1428] Setting key 30bfa7b311b03c1de25f969b12c482e4
I20260926 05:41:54.366761 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:41:54.369040 29086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.368063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38304 (local address 127.23.213.254:38315)
0926 05:41:54.368119 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.368121 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.368205 (+ 84us) server_negotiation.cc:407] Connection header received
0926 05:41:54.368250 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.368251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.368255 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.368302 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.368894 (+ 592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.368895 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.368895 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.368898 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.368910 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.368912 (+ 2us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.368950 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.368983 (+ 33us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.368994 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:41:54.370867 29029 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:38304:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:41:54.372499 29029 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:41:54.376250 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.374552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:37381 (local address 127.0.0.1:42636)
0926 05:41:54.374841 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.374848 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.374870 (+ 22us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.375418 (+ 548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.375419 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.375421 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.375842 (+ 421us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.375842 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.375846 (+ 4us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.375854 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.376085 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.376090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.376093 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.376161 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.376175 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":90,"threads_started":1}
I20260926 05:41:54.376269 29489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.374626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42636 (local address 127.23.213.195:37381)
0926 05:41:54.375292 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.375296 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.375306 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:41:54.375329 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.375331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.375341 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.375404 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.375926 (+ 522us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.375932 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.375933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.375937 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.375975 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.376001 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.376099 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.376171 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.376185 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":629,"thread_start_us":84,"threads_started":1}
I20260926 05:41:54.376461 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.374815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.23.213.194:36019)
0926 05:41:54.374985 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.374990 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.375005 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:41:54.375040 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.375043 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.375052 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.375134 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.376087 (+ 953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.376090 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.376090 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.376093 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.376121 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.376135 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.376316 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.376341 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.376358 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":71,"threads_started":1}
I20260926 05:41:54.376726 29487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.374498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:36019 (local address 127.0.0.1:58612)
0926 05:41:54.374654 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.374696 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.374753 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.375155 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.375162 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.375168 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.375666 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.375667 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.375681 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.375703 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.376228 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.376229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.376229 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.376261 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.376275 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":52,"threads_started":1}
I20260926 05:41:54.376881 29490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.374697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:40231 (local address 127.0.0.1:58162)
0926 05:41:54.375090 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.375100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.375175 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.375626 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.375627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.375630 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.375997 (+ 367us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.375997 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.376002 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.376010 (+ 8us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.376228 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.376229 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.376229 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.376262 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.376276 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":54,"threads_started":1}
I20260926 05:41:54.376961 29413 tablet_service.cc:1467] Processing CreateTablet for tablet 99f8144ab1d14a3ca11ec37badea2e56 (DEFAULT_TABLE table=test-table [id=991a779089f44d009319417b1bd25bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:54.377175 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.374961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58162 (local address 127.23.213.193:40231)
0926 05:41:54.375119 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.375123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.375170 (+ 47us) server_negotiation.cc:407] Connection header received
0926 05:41:54.375268 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.375272 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.375282 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.375769 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.376076 (+ 307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.376080 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.376080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.376083 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.376114 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.376128 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.377055 (+ 927us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.377081 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.377099 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":62,"threads_started":1}
I20260926 05:41:54.377357 29413 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f8144ab1d14a3ca11ec37badea2e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:54.377456 29282 tablet_service.cc:1467] Processing CreateTablet for tablet 99f8144ab1d14a3ca11ec37badea2e56 (DEFAULT_TABLE table=test-table [id=991a779089f44d009319417b1bd25bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:54.377835 29282 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f8144ab1d14a3ca11ec37badea2e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:54.377736 29151 tablet_service.cc:1467] Processing CreateTablet for tablet 99f8144ab1d14a3ca11ec37badea2e56 (DEFAULT_TABLE table=test-table [id=991a779089f44d009319417b1bd25bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:41:54.378046 29151 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 99f8144ab1d14a3ca11ec37badea2e56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:54.380012 29493 tablet_bootstrap.cc:492] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Bootstrap starting.
I20260926 05:41:54.380679 29493 tablet_bootstrap.cc:654] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:54.381019 29493 log.cc:824] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:54.381045 29494 tablet_bootstrap.cc:492] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Bootstrap starting.
I20260926 05:41:54.381403 29495 tablet_bootstrap.cc:492] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610: Bootstrap starting.
I20260926 05:41:54.381790 29494 tablet_bootstrap.cc:654] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:54.382050 29495 tablet_bootstrap.cc:654] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:54.382131 29494 log.cc:824] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:54.382364 29493 tablet_bootstrap.cc:492] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: No bootstrap required, opened a new log
I20260926 05:41:54.382354 29495 log.cc:824] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:54.382442 29493 ts_tablet_manager.cc:1397] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:54.382834 29494 tablet_bootstrap.cc:492] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: No bootstrap required, opened a new log
I20260926 05:41:54.382912 29494 ts_tablet_manager.cc:1397] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:41:54.383090 29495 tablet_bootstrap.cc:492] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610: No bootstrap required, opened a new log
I20260926 05:41:54.383153 29495 ts_tablet_manager.cc:1397] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:41:54.384979 29494 raft_consensus.cc:348] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.385128 29494 raft_consensus.cc:374] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:54.385005 29493 raft_consensus.cc:348] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.385169 29494 raft_consensus.cc:729] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fa4d408e9d9439695def5dca99a5797, State: Initialized, Role: FOLLOWER
I20260926 05:41:54.385179 29493 raft_consensus.cc:374] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:54.385215 29493 raft_consensus.cc:729] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7cb6f5748b8457a83829568ef3080c0, State: Initialized, Role: FOLLOWER
I20260926 05:41:54.385293 29494 consensus_queue.cc:260] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.385397 29493 consensus_queue.cc:260] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.385646 29348 heartbeater.cc:503] Master 127.23.213.254:38315 was elected leader, sending a full tablet report...
I20260926 05:41:54.385686 29493 ts_tablet_manager.cc:1428] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 05:41:54.385845 29479 heartbeater.cc:503] Master 127.23.213.254:38315 was elected leader, sending a full tablet report...
I20260926 05:41:54.385879 29494 ts_tablet_manager.cc:1428] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:41:54.385682 29495 raft_consensus.cc:348] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.386134 29495 raft_consensus.cc:374] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:54.386200 29495 raft_consensus.cc:729] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da19530eeab43e6855a416c269a9610, State: Initialized, Role: FOLLOWER
I20260926 05:41:54.386507 29495 consensus_queue.cc:260] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.386755 29495 ts_tablet_manager.cc:1428] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20260926 05:41:54.386785 29217 heartbeater.cc:503] Master 127.23.213.254:38315 was elected leader, sending a full tablet report...
W20260926 05:41:54.495875 29349 tablet.cc:2369] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:41:54.609540 29480 tablet.cc:2369] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:41:54.630352 29218 tablet.cc:2369] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:41:54.665599 29501 raft_consensus.cc:482] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:41:54.665784 29501 raft_consensus.cc:504] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.666220 29501 leader_election.cc:288] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fa4d408e9d9439695def5dca99a5797 (127.23.213.194:36019), d7cb6f5748b8457a83829568ef3080c0 (127.23.213.195:37381)
I20260926 05:41:54.667708 29491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.666615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:33577 (local address 127.23.213.194:36019)
0926 05:41:54.666673 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.666675 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.666705 (+ 30us) server_negotiation.cc:407] Connection header received
0926 05:41:54.666776 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.666777 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.666783 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.666861 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.667473 (+ 612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.667474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.667475 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.667480 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.667498 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.667503 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.667562 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.667618 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.667634 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:41:54.667871 29502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.666412 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:36019 (local address 127.23.213.193:33577)
0926 05:41:54.666642 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.666654 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.666717 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.666864 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.666865 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.666867 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.667356 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.667357 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.667365 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.667380 (+ 15us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.667557 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.667558 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.667559 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.667618 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.667634 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":84,"threads_started":1}
I20260926 05:41:54.668314 29302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f8144ab1d14a3ca11ec37badea2e56" candidate_uuid: "5da19530eeab43e6855a416c269a9610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa4d408e9d9439695def5dca99a5797" is_pre_election: true
I20260926 05:41:54.668485 29302 raft_consensus.cc:2454] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da19530eeab43e6855a416c269a9610 in term 0.
I20260926 05:41:54.668728 29107 leader_election.cc:302] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa4d408e9d9439695def5dca99a5797, 5da19530eeab43e6855a416c269a9610; no voters:
I20260926 05:41:54.668857 29489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.666554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:37905 (local address 127.23.213.195:37381)
0926 05:41:54.666778 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.666780 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.666882 (+ 102us) server_negotiation.cc:407] Connection header received
0926 05:41:54.666968 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.666969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.666973 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.667029 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.667627 (+ 598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.667629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.667629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.667634 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.667651 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.667656 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.667701 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.668777 (+ 1076us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.668793 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260926 05:41:54.668871 29503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.666466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:37381 (local address 127.23.213.193:37905)
0926 05:41:54.666824 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.666835 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.666902 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.667060 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.667061 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.667074 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:54.667531 (+ 457us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:41:54.667532 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:54.667537 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:41:54.667548 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:54.668698 (+ 1150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:54.668699 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:54.668700 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:41:54.668774 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:41:54.668791 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":62,"threads_started":1}
I20260926 05:41:54.668962 29501 raft_consensus.cc:2781] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:41:54.669023 29501 raft_consensus.cc:482] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:41:54.669040 29501 raft_consensus.cc:3037] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:54.669270 29433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f8144ab1d14a3ca11ec37badea2e56" candidate_uuid: "5da19530eeab43e6855a416c269a9610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cb6f5748b8457a83829568ef3080c0" is_pre_election: true
I20260926 05:41:54.669414 29433 raft_consensus.cc:2454] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5da19530eeab43e6855a416c269a9610 in term 0.
I20260926 05:41:54.669981 29501 raft_consensus.cc:504] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.670125 29501 leader_election.cc:288] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [CANDIDATE]: Term 1 election: Requested vote from peers 2fa4d408e9d9439695def5dca99a5797 (127.23.213.194:36019), d7cb6f5748b8457a83829568ef3080c0 (127.23.213.195:37381)
I20260926 05:41:54.670282 29302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f8144ab1d14a3ca11ec37badea2e56" candidate_uuid: "5da19530eeab43e6855a416c269a9610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fa4d408e9d9439695def5dca99a5797"
I20260926 05:41:54.670334 29433 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "99f8144ab1d14a3ca11ec37badea2e56" candidate_uuid: "5da19530eeab43e6855a416c269a9610" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7cb6f5748b8457a83829568ef3080c0"
I20260926 05:41:54.670383 29302 raft_consensus.cc:3037] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:54.670444 29433 raft_consensus.cc:3037] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:54.671144 29302 raft_consensus.cc:2454] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da19530eeab43e6855a416c269a9610 in term 1.
I20260926 05:41:54.671160 29433 raft_consensus.cc:2454] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5da19530eeab43e6855a416c269a9610 in term 1.
I20260926 05:41:54.671311 29107 leader_election.cc:302] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fa4d408e9d9439695def5dca99a5797, 5da19530eeab43e6855a416c269a9610; no voters:
I20260926 05:41:54.671417 29501 raft_consensus.cc:2781] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:54.671566 29501 raft_consensus.cc:686] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [term 1 LEADER]: Becoming Leader. State: Replica: 5da19530eeab43e6855a416c269a9610, State: Running, Role: LEADER
I20260926 05:41:54.671665 29501 consensus_queue.cc:237] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } }
I20260926 05:41:54.672555 29028 catalog_manager.cc:5495] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5da19530eeab43e6855a416c269a9610 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "5da19530eeab43e6855a416c269a9610" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5da19530eeab43e6855a416c269a9610" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 40231 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 } health_report { overall_health: UNKNOWN } } }
I20260926 05:41:54.734261 29492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.733115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58172 (local address 127.23.213.193:40231)
0926 05:41:54.733181 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.733183 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.733192 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:41:54.733241 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.733244 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.733263 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.733327 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.734015 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.734016 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.734016 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.734020 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.734037 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.734042 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.734117 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.734164 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.734180 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:41:54.736195 29433 raft_consensus.cc:1261] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 1 FOLLOWER]: Refusing update from remote peer 5da19530eeab43e6855a416c269a9610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:54.736526 29302 raft_consensus.cc:1261] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 1 FOLLOWER]: Refusing update from remote peer 5da19530eeab43e6855a416c269a9610: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:41:54.736577 29501 consensus_queue.cc:1035] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7cb6f5748b8457a83829568ef3080c0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 37381 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:54.736814 29504 consensus_queue.cc:1035] T 99f8144ab1d14a3ca11ec37badea2e56 P 5da19530eeab43e6855a416c269a9610 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fa4d408e9d9439695def5dca99a5797" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36019 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:41:54.741214 29028 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:38304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:54.741335 29028 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:38304:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:41:54.742149 29028 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 81a12daec7e14ab792da5502010f29ea: Sending DeleteTablet for 3 replicas of tablet 99f8144ab1d14a3ca11ec37badea2e56
I20260926 05:41:54.742408 29282 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f8144ab1d14a3ca11ec37badea2e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:54 UTC) from {username='slave'} at 127.0.0.1:58612
I20260926 05:41:54.742511 29150 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f8144ab1d14a3ca11ec37badea2e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:54 UTC) from {username='slave'} at 127.0.0.1:58162
I20260926 05:41:54.742568 29413 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f8144ab1d14a3ca11ec37badea2e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:54 UTC) from {username='slave'} at 127.0.0.1:42636
I20260926 05:41:54.742622 29151 tablet_service.cc:1514] Processing DeleteTablet for tablet 99f8144ab1d14a3ca11ec37badea2e56 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:41:54 UTC) from {username='slave'} at 127.0.0.1:58162
I20260926 05:41:54.742856 29517 tablet_replica.cc:330] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: stopping tablet replica
I20260926 05:41:54.742895 29519 tablet_replica.cc:330] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: stopping tablet replica
I20260926 05:41:54.743011 29519 raft_consensus.cc:2229] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:54.743091 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29089
I20260926 05:41:54.743129 29519 raft_consensus.cc:2258] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:54.743556 29519 ts_tablet_manager.cc:1905] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:54.744323 29517 raft_consensus.cc:2229] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:54.744459 29517 raft_consensus.cc:2258] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:54.744876 29517 ts_tablet_manager.cc:1905] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:54.745507 29519 ts_tablet_manager.cc:1918] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:41:54.745586 29519 log.cc:1196] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/99f8144ab1d14a3ca11ec37badea2e56
I20260926 05:41:54.745878 29519 ts_tablet_manager.cc:1939] T 99f8144ab1d14a3ca11ec37badea2e56 P d7cb6f5748b8457a83829568ef3080c0: Deleting consensus metadata
I20260926 05:41:54.746317 29013 catalog_manager.cc:4841] TS d7cb6f5748b8457a83829568ef3080c0 (127.23.213.195:37381): tablet 99f8144ab1d14a3ca11ec37badea2e56 (table test-table [id=991a779089f44d009319417b1bd25bbc]) successfully deleted
I20260926 05:41:54.746381 29517 ts_tablet_manager.cc:1918] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:41:54.746438 29517 log.cc:1196] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/99f8144ab1d14a3ca11ec37badea2e56
I20260926 05:41:54.746673 29517 ts_tablet_manager.cc:1939] T 99f8144ab1d14a3ca11ec37badea2e56 P 2fa4d408e9d9439695def5dca99a5797: Deleting consensus metadata
I20260926 05:41:54.747080 29016 catalog_manager.cc:4841] TS 2fa4d408e9d9439695def5dca99a5797 (127.23.213.194:36019): tablet 99f8144ab1d14a3ca11ec37badea2e56 (table test-table [id=991a779089f44d009319417b1bd25bbc]) successfully deleted
W20260926 05:41:54.749856 29016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:40231 (error 108)
I20260926 05:41:54.750097 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29220
I20260926 05:41:54.750671 29488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.750203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:40231 (local address 127.0.0.1:58178)
0926 05:41:54.750588 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.750633 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:40231: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362}
W20260926 05:41:54.753898 29016 catalog_manager.cc:4584] TS 5da19530eeab43e6855a416c269a9610 (127.23.213.193:40231): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 99f8144ab1d14a3ca11ec37badea2e56: Network error: Client connection negotiation failed: client connection to 127.23.213.193:40231: connect: Connection refused (error 111)
I20260926 05:41:54.755362 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29351
I20260926 05:41:54.760660 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 28998
2026-09-26T05:41:54Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (915 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 05:41:54.780614 24407 mini_oidc.cc:152] Starting JWKS server
I20260926 05:41:54.781715 24407 webserver.cc:460] Webserver started at https://127.0.0.1:41645/ using document root <none> and password file <none>
I20260926 05:41:54.781879 24407 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:41:54.782056 24407 webserver.cc:460] Webserver started at http://127.0.0.1:41619/ using document root <none> and password file <none>
2026-09-26T05:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:54Z Disabled control of system clock
I20260926 05:41:54.790104 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:44113
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:37593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:44113
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41645/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/testchainca.cert with env {}
W20260926 05:41:54.870754 29526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:54.870973 29526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:54.870996 29526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:54.872125 29526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:54.872160 29526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:54.872174 29526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:54.872185 29526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:54.872197 29526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:41:54.872210 29526 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:41:54.872220 29526 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41645/jwks.json
I20260926 05:41:54.873718 29526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:44113
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:44113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41645/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:54.874014 29526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:54.874183 29526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:54.879249 29532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:54.879282 29531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:54.879277 29534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:54.879318 29526 server_base.cc:1034] running on GCE node
I20260926 05:41:54.879680 29526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:54.880016 29526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:54.881156 29526 hybrid_clock.cc:648] HybridClock initialized: now 1790401314881139 us; error 43 us; skew 500 ppm
I20260926 05:41:54.882431 29526 webserver.cc:460] Webserver started at http://127.23.213.254:35475/ using document root <none> and password file <none>
I20260926 05:41:54.882686 29526 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:54.882735 29526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:54.882874 29526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:54.883993 29526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "955543519986490f968eaaba9898121c"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "33eda033f42d3308c8124cd7c312a58e"
server_key_iv: "8bf24dadc98cc9a93d48d54ea93e55bf"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.884411 29526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "955543519986490f968eaaba9898121c"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "33eda033f42d3308c8124cd7c312a58e"
server_key_iv: "8bf24dadc98cc9a93d48d54ea93e55bf"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.885725 29526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:54.886551 29540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:54.886715 29526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:54.886816 29526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "955543519986490f968eaaba9898121c"
format_stamp: "Formatted at 2026-09-26 05:41:54 on dist-test-slave-1fdt"
server_key: "33eda033f42d3308c8124cd7c312a58e"
server_key_iv: "8bf24dadc98cc9a93d48d54ea93e55bf"
server_key_version: "encryptionkey@0"
I20260926 05:41:54.886896 29526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:54.903982 29526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:54.904372 29526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:54.904534 29526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:54.908751 29526 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:44113
I20260926 05:41:54.908785 29592 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:44113 every 8 connection(s)
I20260926 05:41:54.909154 29526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:54.910236 29593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:54.912647 29593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c: Bootstrap starting.
I20260926 05:41:54.913285 29593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:54.913604 29593 log.cc:824] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:54.914307 29593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c: No bootstrap required, opened a new log
I20260926 05:41:54.914561 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29526
I20260926 05:41:54.914665 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:54.914844 24407 external_mini_cluster.cc:1428] Setting key 19c78a19de071922e23866fde9388fa4
I20260926 05:41:54.916407 29593 raft_consensus.cc:348] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955543519986490f968eaaba9898121c" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44113 } }
I20260926 05:41:54.916615 29593 raft_consensus.cc:374] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:54.916662 29593 raft_consensus.cc:729] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 955543519986490f968eaaba9898121c, State: Initialized, Role: FOLLOWER
I20260926 05:41:54.916797 29593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955543519986490f968eaaba9898121c" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44113 } }
I20260926 05:41:54.916891 29593 raft_consensus.cc:388] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:54.916916 29593 raft_consensus.cc:482] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:54.916985 29593 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:54.917537 29593 raft_consensus.cc:504] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955543519986490f968eaaba9898121c" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44113 } }
I20260926 05:41:54.917623 29593 leader_election.cc:302] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 955543519986490f968eaaba9898121c; no voters:
I20260926 05:41:54.917819 29593 leader_election.cc:288] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:54.917848 29598 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:54.917951 29598 raft_consensus.cc:686] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [term 1 LEADER]: Becoming Leader. State: Replica: 955543519986490f968eaaba9898121c, State: Running, Role: LEADER
I20260926 05:41:54.918051 29598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955543519986490f968eaaba9898121c" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44113 } }
I20260926 05:41:54.918092 29593 sys_catalog.cc:564] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:54.918552 29598 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 955543519986490f968eaaba9898121c. Latest consensus state: current_term: 1 leader_uuid: "955543519986490f968eaaba9898121c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955543519986490f968eaaba9898121c" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44113 } } }
I20260926 05:41:54.918629 29598 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:54.918776 29599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "955543519986490f968eaaba9898121c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "955543519986490f968eaaba9898121c" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44113 } } }
I20260926 05:41:54.918859 29599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:54.918953 29605 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:54.919487 29605 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:54.921173 29605 catalog_manager.cc:1347] Generated new cluster ID: e418427b1d5549a8b1ed1695a919b806
I20260926 05:41:54.921229 29605 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:54.921712 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.915380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48748 (local address 127.23.213.254:44113)
0926 05:41:54.915583 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.915592 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.915623 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:41:54.916377 (+ 754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.916384 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.916454 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.916643 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:54.916838 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:54.918475 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:54.919612 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:54.920112 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:54.920255 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:54.920943 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:54.920946 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:54.920947 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:54.921147 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:54.921191 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:54.921207 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:54.921273 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:54.921377 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 05:41:54.921544 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":83,"threads_started":1}
I20260926 05:41:54.922755 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:41:54.936407 29605 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:54.936957 29605 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:54.940480 29605 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 955543519986490f968eaaba9898121c: Generated new TSK 0
I20260926 05:41:54.940685 29605 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 05:41:54.961090 29623 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 05:41:54.961205 29623 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 05:41:54.961465 29597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:54.946262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48756 (local address 127.23.213.254:44113)
0926 05:41:54.946304 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:54.946306 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:54.946426 (+ 120us) server_negotiation.cc:407] Connection header received
0926 05:41:54.946531 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:54.946532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:54.946541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:54.946601 (+ 60us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:41:54.946762 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:54.947247 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:54.948379 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:54.948552 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:54.948598 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:54.948826 (+ 228us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:41:54.961237 (+ 12411us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 05:41:54.961413 (+ 176us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:41:54.961511 29597 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48756: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 05:41:54.962090 29622 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:54.946157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:44113 (local address 127.0.0.1:48756)
0926 05:41:54.946351 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:54.946363 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:54.946438 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:54.946627 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:54.946628 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:54.946630 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:41:54.946699 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:54.946701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:54.947325 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:54.947326 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:54.948289 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:54.948292 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:54.948622 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:54.948623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:54.948691 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:54.948713 (+ 22us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:41:54.961654 (+ 12941us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 05:41:54.961836 (+ 182us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.213.254:44113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.23.213.254:44113: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41645/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 05:41:54.962667 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29526
2026-09-26T05:41:54Z chronyd exiting
I20260926 05:41:54.998812 24407 test_util.cc:175] -----------------------------------------------
I20260926 05:41:54.998898 24407 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790401292175461-24407-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (219 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 05:41:54.999678 24407 mini_oidc.cc:152] Starting JWKS server
I20260926 05:41:55.000391 24407 webserver.cc:460] Webserver started at https://127.0.0.1:40845/ using document root <none> and password file <none>
I20260926 05:41:55.000465 24407 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:41:55.000648 24407 webserver.cc:460] Webserver started at http://127.0.0.1:43053/ using document root <none> and password file <none>
2026-09-26T05:41:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:54Z Disabled control of system clock
I20260926 05:41:55.008251 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33001
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:40121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:33001
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40845/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/testchainca.cert with env {}
W20260926 05:41:55.092262 29629 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:55.092460 29629 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:55.092483 29629 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:55.093659 29629 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:55.093699 29629 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:55.093712 29629 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:55.093724 29629 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:55.093814 29629 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:41:55.093859 29629 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:41:55.093889 29629 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40845/jwks.json
I20260926 05:41:55.095433 29629 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:40121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:33001
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40845/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29629
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:55.095690 29629 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:55.095906 29629 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:55.100843 29635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:55.100876 29637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:55.100873 29634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:55.101046 29629 server_base.cc:1034] running on GCE node
I20260926 05:41:55.101336 29629 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:55.101660 29629 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:55.102808 29629 hybrid_clock.cc:648] HybridClock initialized: now 1790401315102788 us; error 40 us; skew 500 ppm
I20260926 05:41:55.104117 29629 webserver.cc:460] Webserver started at http://127.23.213.254:43779/ using document root <none> and password file <none>
I20260926 05:41:55.104372 29629 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:55.104418 29629 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:55.104548 29629 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:55.105618 29629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "296a4924f7ac42ac983f39cc6b4494b7"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "e38cd1b03a8e0b776dcd63afbdaed17c"
server_key_iv: "adbc2266849d43965c2ac8bc7e521d12"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.106040 29629 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "296a4924f7ac42ac983f39cc6b4494b7"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "e38cd1b03a8e0b776dcd63afbdaed17c"
server_key_iv: "adbc2266849d43965c2ac8bc7e521d12"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.107337 29629 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:55.108104 29643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:55.108319 29629 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:55.108413 29629 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "296a4924f7ac42ac983f39cc6b4494b7"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "e38cd1b03a8e0b776dcd63afbdaed17c"
server_key_iv: "adbc2266849d43965c2ac8bc7e521d12"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.108492 29629 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:55.125218 29629 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:55.125625 29629 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:55.125823 29629 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:55.129781 29629 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:33001
I20260926 05:41:55.129805 29695 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:33001 every 8 connection(s)
I20260926 05:41:55.130154 29629 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:55.130851 29696 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:55.132423 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29629
I20260926 05:41:55.132534 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:55.132711 24407 external_mini_cluster.cc:1428] Setting key c9a6fb9a10a4215d47e749859784fb56
I20260926 05:41:55.134209 29696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7: Bootstrap starting.
I20260926 05:41:55.134887 29696 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:55.135233 29696 log.cc:824] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:55.136020 29696 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7: No bootstrap required, opened a new log
I20260926 05:41:55.138379 29696 raft_consensus.cc:348] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a4924f7ac42ac983f39cc6b4494b7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33001 } }
I20260926 05:41:55.138549 29696 raft_consensus.cc:374] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:55.138576 29696 raft_consensus.cc:729] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 296a4924f7ac42ac983f39cc6b4494b7, State: Initialized, Role: FOLLOWER
I20260926 05:41:55.138741 29696 consensus_queue.cc:260] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a4924f7ac42ac983f39cc6b4494b7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33001 } }
I20260926 05:41:55.138862 29696 raft_consensus.cc:388] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:55.138931 29696 raft_consensus.cc:482] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:55.138988 29696 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:55.139556 29699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:55.133258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54416 (local address 127.23.213.254:33001)
0926 05:41:55.133458 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:55.133466 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:55.134096 (+ 630us) server_negotiation.cc:407] Connection header received
0926 05:41:55.134600 (+ 504us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:55.134607 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:55.134669 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:55.134785 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:55.135103 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.136482 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.137248 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.137964 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.138117 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.138816 (+ 699us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:55.138819 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:55.138820 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:55.139034 (+ 214us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:55.139078 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:55.139105 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:55.139161 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:55.139279 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 05:41:55.139370 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260926 05:41:55.139722 29696 raft_consensus.cc:504] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a4924f7ac42ac983f39cc6b4494b7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33001 } }
I20260926 05:41:55.139837 29696 leader_election.cc:302] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 296a4924f7ac42ac983f39cc6b4494b7; no voters:
I20260926 05:41:55.140013 29696 leader_election.cc:288] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:55.140055 29701 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:55.140192 29701 raft_consensus.cc:686] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [term 1 LEADER]: Becoming Leader. State: Replica: 296a4924f7ac42ac983f39cc6b4494b7, State: Running, Role: LEADER
I20260926 05:41:55.140277 29696 sys_catalog.cc:564] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:55.140277 29701 consensus_queue.cc:237] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a4924f7ac42ac983f39cc6b4494b7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33001 } }
I20260926 05:41:55.140674 29701 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 296a4924f7ac42ac983f39cc6b4494b7. Latest consensus state: current_term: 1 leader_uuid: "296a4924f7ac42ac983f39cc6b4494b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a4924f7ac42ac983f39cc6b4494b7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33001 } } }
I20260926 05:41:55.140751 29701 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:55.140745 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:41:55.140925 29707 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:55.141541 29707 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:55.142078 29703 sys_catalog.cc:455] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "296a4924f7ac42ac983f39cc6b4494b7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "296a4924f7ac42ac983f39cc6b4494b7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33001 } } }
I20260926 05:41:55.142194 29703 sys_catalog.cc:458] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:55.143110 29707 catalog_manager.cc:1347] Generated new cluster ID: dfe6b56c08d7462cb482082739a82f17
I20260926 05:41:55.143158 29707 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:55.163597 29707 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:55.164388 29707 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:55.170805 29707 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 296a4924f7ac42ac983f39cc6b4494b7: Generated new TSK 0
I20260926 05:41:55.171020 29707 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 05:41:55.203115 29726 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 05:41:55.203213 29726 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 05:41:55.203524 29699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:55.182477 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54420 (local address 127.23.213.254:33001)
0926 05:41:55.182524 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:55.182526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:55.182630 (+ 104us) server_negotiation.cc:407] Connection header received
0926 05:41:55.182712 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:55.182713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:55.182727 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:55.182786 (+ 59us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:41:55.182937 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.183436 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.184581 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.184770 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.184822 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.185038 (+ 216us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:41:55.203269 (+ 18231us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 05:41:55.203434 (+ 165us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 05:41:55.203598 29699 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54420: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260926 05:41:55.203647 29725 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:55.182381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:33001 (local address 127.0.0.1:54420)
0926 05:41:55.182549 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:55.182562 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:55.182635 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:55.182803 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:55.182804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:55.182806 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:41:55.182871 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:55.182873 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.183533 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.183534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:55.184492 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:55.184495 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.184851 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.184852 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:55.184928 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.184935 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:41:55.203403 (+ 18468us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 05:41:55.203511 (+ 108us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.213.254:33001: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40845/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":97,"threads_started":1}
I20260926 05:41:55.203930 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29629
2026-09-26T05:41:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (243 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 05:41:55.243225 24407 mini_oidc.cc:152] Starting JWKS server
I20260926 05:41:55.243925 24407 webserver.cc:460] Webserver started at https://127.0.0.1:38715/ using document root <none> and password file <none>
I20260926 05:41:55.243992 24407 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 05:41:55.244170 24407 webserver.cc:460] Webserver started at http://127.0.0.1:44789/ using document root <none> and password file <none>
2026-09-26T05:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:55Z Disabled control of system clock
I20260926 05:41:55.251752 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38033
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:34751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38033
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38715/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 05:41:55.331254 29732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:55.331513 29732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:55.331553 29732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:55.332770 29732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:55.332844 29732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:55.332867 29732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:55.332888 29732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:55.332917 29732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:41:55.332943 29732 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:41:55.332970 29732 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38715/jwks.json
I20260926 05:41:55.334364 29732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:34751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38033
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38715/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:55.334647 29732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:55.334949 29732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:41:55.340109 29732 server_base.cc:1034] running on GCE node
W20260926 05:41:55.340020 29740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:55.340034 29737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:55.340016 29738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:55.340534 29732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:55.340791 29732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:55.341934 29732 hybrid_clock.cc:648] HybridClock initialized: now 1790401315341917 us; error 41 us; skew 500 ppm
I20260926 05:41:55.343226 29732 webserver.cc:460] Webserver started at http://127.23.213.254:38385/ using document root <none> and password file <none>
I20260926 05:41:55.343420 29732 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:55.343461 29732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:55.343551 29732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:55.344642 29732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "927f7b0d259d4ea2b09371921d1d31e7"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "c26cd39d7535f80aba3f061bab034b60"
server_key_iv: "dff99c0ef8fc4534dafa656fa067dd36"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.345000 29732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "927f7b0d259d4ea2b09371921d1d31e7"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "c26cd39d7535f80aba3f061bab034b60"
server_key_iv: "dff99c0ef8fc4534dafa656fa067dd36"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.346374 29732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:55.347187 29746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:55.347368 29732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:55.347473 29732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "927f7b0d259d4ea2b09371921d1d31e7"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "c26cd39d7535f80aba3f061bab034b60"
server_key_iv: "dff99c0ef8fc4534dafa656fa067dd36"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.347551 29732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:55.373607 29732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:55.374001 29732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:55.374178 29732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:55.378315 29732 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38033
I20260926 05:41:55.378353 29798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38033 every 8 connection(s)
I20260926 05:41:55.378726 29732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:55.379694 29799 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:55.382141 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7: Bootstrap starting.
I20260926 05:41:55.382676 29799 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:55.382922 29799 log.cc:824] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:55.383544 29799 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7: No bootstrap required, opened a new log
I20260926 05:41:55.385103 29799 raft_consensus.cc:348] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927f7b0d259d4ea2b09371921d1d31e7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38033 } }
I20260926 05:41:55.385243 29799 raft_consensus.cc:374] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:55.385265 29799 raft_consensus.cc:729] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 927f7b0d259d4ea2b09371921d1d31e7, State: Initialized, Role: FOLLOWER
I20260926 05:41:55.385366 29799 consensus_queue.cc:260] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927f7b0d259d4ea2b09371921d1d31e7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38033 } }
I20260926 05:41:55.385447 29799 raft_consensus.cc:388] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:55.385478 29799 raft_consensus.cc:482] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:55.385540 29799 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:55.386161 29799 raft_consensus.cc:504] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927f7b0d259d4ea2b09371921d1d31e7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38033 } }
I20260926 05:41:55.386287 29799 leader_election.cc:302] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 927f7b0d259d4ea2b09371921d1d31e7; no voters:
I20260926 05:41:55.386400 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29732
I20260926 05:41:55.386456 29799 leader_election.cc:288] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:55.386516 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:55.386514 29802 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:55.386654 29802 raft_consensus.cc:686] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [term 1 LEADER]: Becoming Leader. State: Replica: 927f7b0d259d4ea2b09371921d1d31e7, State: Running, Role: LEADER
I20260926 05:41:55.386687 24407 external_mini_cluster.cc:1428] Setting key e846f9b75f1fd22090152c318129614a
I20260926 05:41:55.386693 29799 sys_catalog.cc:564] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:55.386765 29802 consensus_queue.cc:237] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927f7b0d259d4ea2b09371921d1d31e7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38033 } }
I20260926 05:41:55.387056 29803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "927f7b0d259d4ea2b09371921d1d31e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927f7b0d259d4ea2b09371921d1d31e7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38033 } } }
I20260926 05:41:55.387137 29803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:55.387398 29804 sys_catalog.cc:455] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 927f7b0d259d4ea2b09371921d1d31e7. Latest consensus state: current_term: 1 leader_uuid: "927f7b0d259d4ea2b09371921d1d31e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "927f7b0d259d4ea2b09371921d1d31e7" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38033 } } }
I20260926 05:41:55.387452 29804 sys_catalog.cc:458] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:55.387483 29809 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:55.388036 29809 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:55.389715 29809 catalog_manager.cc:1347] Generated new cluster ID: 61826a25e241400dbcb5d44b1c744949
I20260926 05:41:55.389802 29809 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:55.393570 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:55.387174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54148 (local address 127.23.213.254:38033)
0926 05:41:55.387362 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:55.387373 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:55.387406 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:41:55.387522 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:55.387529 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:55.387632 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:55.388058 (+ 426us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:55.388178 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.390212 (+ 2034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.391344 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.391952 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.392070 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.392832 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:55.392836 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:55.392837 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:55.393054 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:55.393094 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:55.393108 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:55.393177 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:55.393268 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 05:41:55.393343 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260926 05:41:55.394598 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:41:55.396391 29809 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:55.397073 29809 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:55.403093 29809 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 927f7b0d259d4ea2b09371921d1d31e7: Generated new TSK 0
I20260926 05:41:55.403270 29809 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:41:55.414407 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:55.410340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54160 (local address 127.23.213.254:38033)
0926 05:41:55.410385 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:55.410387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:55.410463 (+ 76us) server_negotiation.cc:407] Connection header received
0926 05:41:55.410553 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:55.410554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:55.410565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:55.410637 (+ 72us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:41:55.410775 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.411548 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.412367 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.412684 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.412751 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.412898 (+ 147us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:41:55.414202 (+ 1304us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 05:41:55.414338 (+ 136us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:54160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 05:41:55.414464 29810 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:54160: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 05:41:55.416355 29828 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:55.410265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38033 (local address 127.0.0.1:54160)
0926 05:41:55.410409 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:55.410420 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:55.410483 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:55.410641 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:55.410641 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:55.410643 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:41:55.410694 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:55.410696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.411642 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.411643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:55.412290 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:55.412292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.412758 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.412759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:55.412802 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.412806 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:41:55.414310 (+ 1504us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 05:41:55.416252 (+ 1942us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.213.254:38033: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38715/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":121,"mutex_wait_us":1771,"thread_start_us":99,"threads_started":1}
I20260926 05:41:55.416682 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29732
E20260926 05:41:55.425647 29829 webserver.cc:533] Webserver: sslize failed:
E20260926 05:41:55.425724 29829 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T05:41:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (210 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T05:41:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:55Z Disabled control of system clock
I20260926 05:41:55.461136 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38863
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:36027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38863
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/testchainca.cert with env {}
W20260926 05:41:55.539628 29833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:55.539870 29833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:55.539906 29833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:55.541067 29833 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:55.541142 29833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:55.541162 29833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:55.541183 29833 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:55.541213 29833 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 05:41:55.541237 29833 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 05:41:55.541262 29833 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 05:41:55.542752 29833 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38863
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38863
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29833
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:55.543020 29833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:55.543232 29833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:55.548234 29841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:55.548254 29839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:55.548237 29838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:55.548522 29833 server_base.cc:1034] running on GCE node
I20260926 05:41:55.548765 29833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:55.549075 29833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:55.550232 29833 hybrid_clock.cc:648] HybridClock initialized: now 1790401315550218 us; error 44 us; skew 500 ppm
I20260926 05:41:55.551386 29833 webserver.cc:460] Webserver started at http://127.23.213.254:34187/ using document root <none> and password file <none>
I20260926 05:41:55.551646 29833 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:55.551714 29833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:55.551863 29833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:55.552930 29833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "b9ef72f323e6451391e9494b3751e439"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "fe306822b99f0ceb9246eaa31f410021"
server_key_iv: "01c4b645679b0a01a2df864019d39c2e"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.553285 29833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "b9ef72f323e6451391e9494b3751e439"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "fe306822b99f0ceb9246eaa31f410021"
server_key_iv: "01c4b645679b0a01a2df864019d39c2e"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.554689 29833 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:55.555475 29847 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:55.555660 29833 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:55.555752 29833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "b9ef72f323e6451391e9494b3751e439"
format_stamp: "Formatted at 2026-09-26 05:41:55 on dist-test-slave-1fdt"
server_key: "fe306822b99f0ceb9246eaa31f410021"
server_key_iv: "01c4b645679b0a01a2df864019d39c2e"
server_key_version: "encryptionkey@0"
I20260926 05:41:55.555831 29833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:55.584540 29833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:55.584895 29833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:55.585055 29833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:55.589213 29833 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38863
I20260926 05:41:55.589255 29899 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38863 every 8 connection(s)
I20260926 05:41:55.589640 29833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:55.590643 29900 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:55.593039 29900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439: Bootstrap starting.
I20260926 05:41:55.593627 29900 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:55.593979 29900 log.cc:824] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:55.594640 29900 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439: No bootstrap required, opened a new log
I20260926 05:41:55.595577 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29833
I20260926 05:41:55.595698 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:55.595901 24407 external_mini_cluster.cc:1428] Setting key d41a420893b526c1b86cc089356b2a0b
I20260926 05:41:55.596884 29900 raft_consensus.cc:348] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef72f323e6451391e9494b3751e439" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38863 } }
I20260926 05:41:55.597050 29900 raft_consensus.cc:374] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:55.597095 29900 raft_consensus.cc:729] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b9ef72f323e6451391e9494b3751e439, State: Initialized, Role: FOLLOWER
I20260926 05:41:55.597223 29900 consensus_queue.cc:260] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef72f323e6451391e9494b3751e439" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38863 } }
I20260926 05:41:55.597328 29900 raft_consensus.cc:388] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:55.597373 29900 raft_consensus.cc:482] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:55.597424 29900 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:55.598042 29900 raft_consensus.cc:504] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef72f323e6451391e9494b3751e439" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38863 } }
I20260926 05:41:55.598163 29900 leader_election.cc:302] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b9ef72f323e6451391e9494b3751e439; no voters:
I20260926 05:41:55.598352 29900 leader_election.cc:288] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:55.598398 29905 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:55.598533 29905 raft_consensus.cc:686] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [term 1 LEADER]: Becoming Leader. State: Replica: b9ef72f323e6451391e9494b3751e439, State: Running, Role: LEADER
I20260926 05:41:55.598620 29900 sys_catalog.cc:564] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:55.598623 29905 consensus_queue.cc:237] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef72f323e6451391e9494b3751e439" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38863 } }
I20260926 05:41:55.600286 29906 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b9ef72f323e6451391e9494b3751e439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef72f323e6451391e9494b3751e439" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38863 } } }
I20260926 05:41:55.600572 29906 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:55.600392 29907 sys_catalog.cc:455] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b9ef72f323e6451391e9494b3751e439. Latest consensus state: current_term: 1 leader_uuid: "b9ef72f323e6451391e9494b3751e439" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b9ef72f323e6451391e9494b3751e439" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38863 } } }
I20260926 05:41:55.600742 29907 sys_catalog.cc:458] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439 [sys.catalog]: This master's current role is: LEADER
W20260926 05:41:55.601257 29919 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:41:55.601326 29919 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:41:55.601384 29921 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:55.601567 29921 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:55.602103 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:55.596474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.23.213.254:38863)
0926 05:41:55.596694 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:55.596701 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:55.596724 (+ 23us) server_negotiation.cc:407] Connection header received
0926 05:41:55.596851 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:55.596857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:55.596909 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:55.597040 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:55.597231 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.598764 (+ 1533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.599797 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.600345 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.600462 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.601359 (+ 897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:55.601364 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:55.601364 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:41:55.601598 (+ 234us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:55.601663 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:41:55.601678 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:55.601764 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:55.601843 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 05:41:55.601923 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":75,"threads_started":1}
I20260926 05:41:55.603049 29921 catalog_manager.cc:1347] Generated new cluster ID: 045b5784b1b1473db68b2d48f8655f17
I20260926 05:41:55.603080 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:41:55.603101 29921 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:55.617631 29921 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:55.618217 29921 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:55.626219 29921 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b9ef72f323e6451391e9494b3751e439: Generated new TSK 0
I20260926 05:41:55.626475 29921 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:41:55.654388 29904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:55.633013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.23.213.254:38863)
0926 05:41:55.633056 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:55.633062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:55.633116 (+ 54us) server_negotiation.cc:407] Connection header received
0926 05:41:55.633205 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:55.633206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:55.633215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:55.633273 (+ 58us) server_negotiation.cc:226] Negotiated authn=JWT
0926 05:41:55.633445 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.633980 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.635121 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.635299 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.635343 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.635553 (+ 210us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 05:41:55.654180 (+ 18627us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 05:41:55.654329 (+ 149us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42196: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 05:41:55.654479 29904 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42196: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 05:41:55.654584 29929 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:55.632897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38863 (local address 127.0.0.1:42196)
0926 05:41:55.633057 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:55.633069 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:55.633136 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:55.633292 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:55.633293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:55.633295 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 05:41:55.633373 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:55.633376 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.634058 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.634059 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:55.635032 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:55.635035 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:55.635369 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:55.635370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:55.635453 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:55.635459 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 05:41:55.654316 (+ 18857us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 05:41:55.654468 (+ 152us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.23.213.254:38863: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260926 05:41:55.654973 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29833
2026-09-26T05:41:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (218 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (88 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (88 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:55 dist-test-slave-1fdt krb5kdc[29941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:55 dist-test-slave-1fdt krb5kdc[29941](info): set up 2 sockets
Sep 26 05:41:55 dist-test-slave-1fdt krb5kdc[29941](info): commencing operation
krb5kdc: starting...
W20260926 05:41:56.862041 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:56 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:41:56.980655 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:39173
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:39173
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:57.062281 29954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:57.062497 29954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:57.062520 29954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:57.063638 29954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:57.063673 29954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:57.063687 29954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:57.063699 29954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:57.063711 29954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:57.065059 29954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:39173
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:39173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:57.065241 29954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:57.065399 29954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:57.070010 29960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:57.070010 29962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:57.070040 29959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:57.070261 29954 server_base.cc:1034] running on GCE node
I20260926 05:41:57.070490 29954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:57.070737 29954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:57.071899 29954 hybrid_clock.cc:648] HybridClock initialized: now 1790401317071888 us; error 29 us; skew 500 ppm
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.077525 29954 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:41:57.078060 29954 webserver.cc:460] Webserver started at http://127.23.213.254:40843/ using document root <none> and password file <none>
I20260926 05:41:57.078286 29954 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:57.078330 29954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:57.078496 29954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:57.079569 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "c96551fdf6d240909632079a791a4d10"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "eb3f8aab40e1e1ea6b5d014cddde750a"
server_key_iv: "eded16c25ba00fdf53c51a10698f5e99"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.079936 29954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "c96551fdf6d240909632079a791a4d10"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "eb3f8aab40e1e1ea6b5d014cddde750a"
server_key_iv: "eded16c25ba00fdf53c51a10698f5e99"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.081234 29954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:57.082114 29969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.082296 29954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.082388 29954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "c96551fdf6d240909632079a791a4d10"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "eb3f8aab40e1e1ea6b5d014cddde750a"
server_key_iv: "eded16c25ba00fdf53c51a10698f5e99"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.082466 29954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:57.096520 29954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:57.098611 29954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:57.098775 29954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:57.102854 29954 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:39173
I20260926 05:41:57.102895 30021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:39173 every 8 connection(s)
I20260926 05:41:57.103262 29954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:57.104107 30022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:57.105463 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 29954
I20260926 05:41:57.105564 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:57.105765 24407 external_mini_cluster.cc:1428] Setting key c115a0816acbcbc041772b66f7f45f20
I20260926 05:41:57.106889 30022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10: Bootstrap starting.
I20260926 05:41:57.107605 30022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:57.107939 30022 log.cc:824] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:57.108721 30022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10: No bootstrap required, opened a new log
I20260926 05:41:57.110875 30022 raft_consensus.cc:348] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96551fdf6d240909632079a791a4d10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39173 } }
I20260926 05:41:57.111058 30022 raft_consensus.cc:374] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:57.111100 30022 raft_consensus.cc:729] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c96551fdf6d240909632079a791a4d10, State: Initialized, Role: FOLLOWER
I20260926 05:41:57.111253 30022 consensus_queue.cc:260] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96551fdf6d240909632079a791a4d10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39173 } }
I20260926 05:41:57.111313 30022 raft_consensus.cc:388] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:57.111393 30022 raft_consensus.cc:482] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:57.111470 30022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:57.112221 30022 raft_consensus.cc:504] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96551fdf6d240909632079a791a4d10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39173 } }
I20260926 05:41:57.112332 30022 leader_election.cc:302] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c96551fdf6d240909632079a791a4d10; no voters:
I20260926 05:41:57.112493 30022 leader_election.cc:288] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:57.112573 30027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:57.112771 30022 sys_catalog.cc:564] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:57.112777 30027 raft_consensus.cc:686] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [term 1 LEADER]: Becoming Leader. State: Replica: c96551fdf6d240909632079a791a4d10, State: Running, Role: LEADER
I20260926 05:41:57.113018 30027 consensus_queue.cc:237] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96551fdf6d240909632079a791a4d10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39173 } }
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401316, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:57.113322 30027 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c96551fdf6d240909632079a791a4d10. Latest consensus state: current_term: 1 leader_uuid: "c96551fdf6d240909632079a791a4d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96551fdf6d240909632079a791a4d10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39173 } } }
I20260926 05:41:57.113401 30027 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:57.113351 30029 sys_catalog.cc:455] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c96551fdf6d240909632079a791a4d10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c96551fdf6d240909632079a791a4d10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39173 } } }
I20260926 05:41:57.113538 30029 sys_catalog.cc:458] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10 [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:57.113699 30035 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:57.114464 30035 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:57.116076 30035 catalog_manager.cc:1347] Generated new cluster ID: 11f492d6672b44119588d38030f6d53d
I20260926 05:41:57.116128 30035 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:41:57.117790 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.106317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58104 (local address 127.23.213.254:39173)
0926 05:41:57.106507 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.106516 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.106553 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:41:57.106964 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.106971 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.107058 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.107204 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.108003 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.109251 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.110337 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.110756 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.110844 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.113921 (+ 3077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:57.113926 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:57.113927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:57.113941 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:57.116019 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.116456 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.116459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.116459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.116573 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.116808 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.116808 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.116809 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.117110 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:57.117236 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:57.117392 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 05:41:57.117529 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260926 05:41:57.141491 30035 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:57.142342 30035 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:57.155578 30035 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c96551fdf6d240909632079a791a4d10: Generated new TSK 0
I20260926 05:41:57.155822 30035 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:41:57.184762 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39173
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:57.273434 30048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:57.273695 30048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:57.273736 30048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:57.274928 30048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:57.275007 30048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:57.275056 30048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:57.276594 30048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:57.276952 30048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:57.277215 30048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:57.282225 30056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:57.282262 30053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:57.282269 30054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:57.282467 30048 server_base.cc:1034] running on GCE node
I20260926 05:41:57.282662 30048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:57.282951 30048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:57.284137 30048 hybrid_clock.cc:648] HybridClock initialized: now 1790401317284101 us; error 39 us; skew 500 ppm
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.286892 30048 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:41:57.287331 30048 webserver.cc:460] Webserver started at http://127.23.213.193:42729/ using document root <none> and password file <none>
I20260926 05:41:57.287513 30048 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:57.287554 30048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:57.287642 30048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:57.288651 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "046d8bd43abb47d3b6c2b3a7f1b29010"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "0d64a58e8a8c8b8bc1e4d25293bb451d"
server_key_iv: "db9e6ed646a1d558a78cd0cecaab7e33"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.288965 30048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "046d8bd43abb47d3b6c2b3a7f1b29010"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "0d64a58e8a8c8b8bc1e4d25293bb451d"
server_key_iv: "db9e6ed646a1d558a78cd0cecaab7e33"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.290309 30048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:57.291047 30063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.291235 30048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:57.291314 30048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "046d8bd43abb47d3b6c2b3a7f1b29010"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "0d64a58e8a8c8b8bc1e4d25293bb451d"
server_key_iv: "db9e6ed646a1d558a78cd0cecaab7e33"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.291368 30048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:57.305438 30048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:57.307492 30048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:57.307601 30048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:57.307777 30048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:57.308089 30048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:57.308122 30048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.308143 30048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:57.308192 30048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.314200 30048 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:39899
I20260926 05:41:57.314293 30176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:39899 every 8 connection(s)
I20260926 05:41:57.314610 30048 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:57.320230 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30048
I20260926 05:41:57.320345 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:57.320513 24407 external_mini_cluster.cc:1428] Setting key 274e8fa4a0a6a1a1ebcef878b9916f37
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:57.324652 30179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.315410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39173 (local address 127.23.213.193:60973)
0926 05:41:57.315587 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:57.315623 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:57.316489 (+ 866us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:57.316721 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:57.316724 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:57.316957 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:57.317146 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:57.317151 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.317841 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.317842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:57.318644 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:57.318646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.319030 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.319031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:57.319131 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.319593 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:57.319600 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:57.321250 (+ 1650us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:57.323329 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:57.323335 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:57.323337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:57.323544 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:57.323785 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:57.323785 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:57.323785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:57.323880 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:57.324163 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:57.324167 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:57.324278 (+ 111us) client_negotiation.cc:769] Sending connection context
0926 05:41:57.324359 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:41:57.324477 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":115,"threads_started":1}
I20260926 05:41:57.324669 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.315433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:60973 (local address 127.23.213.254:39173)
0926 05:41:57.315501 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.315502 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.315713 (+ 211us) server_negotiation.cc:407] Connection header received
0926 05:41:57.316601 (+ 888us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.316603 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.316614 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.316669 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.317228 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.317776 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.318732 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.318954 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.319012 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.321338 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:57.321339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:57.321340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:57.321349 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:57.323238 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.323635 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.323636 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.323637 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.323707 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.323929 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.323930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.323930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.324090 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:57.324172 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:57.324410 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:41:57.324515 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:41:57.325215 30177 heartbeater.cc:347] Connected to a master server at 127.23.213.254:39173
I20260926 05:41:57.325335 30177 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:57.325564 30177 heartbeater.cc:511] Master 127.23.213.254:39173 requested a full tablet report, sending...
I20260926 05:41:57.326174 29986 ts_manager.cc:194] Registered new tserver with Master: 046d8bd43abb47d3b6c2b3a7f1b29010 (127.23.213.193:39899)
I20260926 05:41:57.326779 29986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:60973
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:41:57.348959 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39173
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:57.432021 30182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:57.432252 30182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:57.432291 30182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:57.433612 30182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:57.433663 30182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:57.433705 30182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:57.435493 30182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:57.435856 30182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:57.436110 30182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:57.441018 30188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:57.441087 30187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:57.441256 30182 server_base.cc:1034] running on GCE node
W20260926 05:41:57.441255 30190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:57.441542 30182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:57.441818 30182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:57.442981 30182 hybrid_clock.cc:648] HybridClock initialized: now 1790401317442960 us; error 41 us; skew 500 ppm
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.445587 30182 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:41:57.446038 30182 webserver.cc:460] Webserver started at http://127.23.213.194:41029/ using document root <none> and password file <none>
I20260926 05:41:57.446285 30182 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:57.446354 30182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:57.446503 30182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:57.447533 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "b6347e572f8144049d7a2ec7deb52298"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "5ba8c781107bf29914973df27fa072b6"
server_key_iv: "14712f6439d444112016fe826cfa24fd"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.447880 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "b6347e572f8144049d7a2ec7deb52298"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "5ba8c781107bf29914973df27fa072b6"
server_key_iv: "14712f6439d444112016fe826cfa24fd"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.449194 30182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:57.449992 30197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.450176 30182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:57.450277 30182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "b6347e572f8144049d7a2ec7deb52298"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "5ba8c781107bf29914973df27fa072b6"
server_key_iv: "14712f6439d444112016fe826cfa24fd"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.450349 30182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:57.467036 30182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:57.468976 30182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:57.469141 30182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:57.469390 30182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:57.469693 30182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:57.469748 30182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.469810 30182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:57.469838 30182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.475391 30182 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:35493
I20260926 05:41:57.475443 30310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:35493 every 8 connection(s)
I20260926 05:41:57.475767 30182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:57.484180 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30182
I20260926 05:41:57.484290 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:57.484513 24407 external_mini_cluster.cc:1428] Setting key 7182edab3a51d8b33ebd17d8558a589c
I20260926 05:41:57.484818 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.476443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:53281 (local address 127.23.213.254:39173)
0926 05:41:57.476483 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.476485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.476693 (+ 208us) server_negotiation.cc:407] Connection header received
0926 05:41:57.477588 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.477589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.477599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.477650 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.478283 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.478768 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.479712 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.479900 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.479966 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.482010 (+ 2044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:57.482012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:57.482012 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:57.482019 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:57.483415 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.483764 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.483765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.483765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.483822 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.484058 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.484058 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.484059 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.484158 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:57.484209 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:57.484610 (+ 401us) server_negotiation.cc:299] Negotiation successful
0926 05:41:57.484706 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:41:57.485008 30313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.476428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39173 (local address 127.23.213.194:53281)
0926 05:41:57.476560 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:57.476597 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:57.477494 (+ 897us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:57.477704 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:57.477708 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:57.478011 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:57.478205 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:57.478210 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.478831 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.478832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:57.479629 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:57.479631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.479974 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.479975 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:57.480071 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.480506 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:57.480514 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:57.481927 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:57.483482 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:57.483486 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:57.483488 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:57.483701 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:57.483877 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:57.483878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:57.483878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:57.483968 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:57.484221 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:57.484227 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:57.484416 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:41:57.484562 (+ 146us) client_negotiation.cc:240] Negotiation successful
0926 05:41:57.484749 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260926 05:41:57.485400 30311 heartbeater.cc:347] Connected to a master server at 127.23.213.254:39173
I20260926 05:41:57.485474 30311 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:57.485662 30311 heartbeater.cc:511] Master 127.23.213.254:39173 requested a full tablet report, sending...
I20260926 05:41:57.486032 29986 ts_manager.cc:194] Registered new tserver with Master: b6347e572f8144049d7a2ec7deb52298 (127.23.213.194:35493)
I20260926 05:41:57.486359 29986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:53281
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:41:57.508400 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39173
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:40037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:57.588747 30316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:57.588949 30316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:57.588971 30316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:57.590196 30316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:57.590236 30316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:57.590262 30316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:57.591678 30316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:40037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:39173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:57.591866 30316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:57.592042 30316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:57.597252 30324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:57.597494 30322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:57.597597 30321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:57.597500 30316 server_base.cc:1034] running on GCE node
I20260926 05:41:57.597880 30316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:57.598132 30316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:57.599305 30316 hybrid_clock.cc:648] HybridClock initialized: now 1790401317599290 us; error 39 us; skew 500 ppm
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.601823 30316 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:41:57.602254 30316 webserver.cc:460] Webserver started at http://127.23.213.195:34627/ using document root <none> and password file <none>
I20260926 05:41:57.602499 30316 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:57.602566 30316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:57.602715 30316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:57.603758 30316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "c0e8f6d78dc944d4b9311ef53c06817d"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "ebd71e64f60da46bfced0f143b1c0d36"
server_key_iv: "f52f3c32f2b185752d73d53184b5484f"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.604105 30316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "c0e8f6d78dc944d4b9311ef53c06817d"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "ebd71e64f60da46bfced0f143b1c0d36"
server_key_iv: "f52f3c32f2b185752d73d53184b5484f"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.605456 30316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:41:57.606264 30331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.606441 30316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.606535 30316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "c0e8f6d78dc944d4b9311ef53c06817d"
format_stamp: "Formatted at 2026-09-26 05:41:57 on dist-test-slave-1fdt"
server_key: "ebd71e64f60da46bfced0f143b1c0d36"
server_key_iv: "f52f3c32f2b185752d73d53184b5484f"
server_key_version: "encryptionkey@0"
I20260926 05:41:57.606637 30316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:57.630235 30316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:57.632267 30316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:57.632427 30316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:57.632673 30316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:57.632982 30316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:57.633033 30316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.633077 30316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:57.633100 30316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:57.638844 30316 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:34517
I20260926 05:41:57.638886 30444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:34517 every 8 connection(s)
I20260926 05:41:57.639220 30316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:57.643435 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30316
I20260926 05:41:57.643522 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:57.643735 24407 external_mini_cluster.cc:1428] Setting key c1fd344edc278e41d6c7253e1136271c
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:41:57.648612 30025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.639957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:52749 (local address 127.23.213.254:39173)
0926 05:41:57.640044 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.640047 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.640215 (+ 168us) server_negotiation.cc:407] Connection header received
0926 05:41:57.641104 (+ 889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.641105 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.641116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.641169 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.641802 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.642307 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.643281 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.643586 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.643673 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.646030 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:57.646032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:57.646032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:57.646039 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:57.647468 (+ 1429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.647854 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.647855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.647855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.647904 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.648108 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.648109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.648109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.648207 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:57.648256 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:57.648473 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 05:41:57.648531 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 05:41:57.648823 30447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.639924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39173 (local address 127.23.213.195:52749)
0926 05:41:57.640090 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:57.640127 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:57.641015 (+ 888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:57.641224 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:57.641228 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:57.641483 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:57.641688 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:57.641692 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.642383 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.642383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:57.643179 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:57.643181 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.643679 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.643680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:57.643783 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.644383 (+ 600us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:57.644394 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:57.645952 (+ 1558us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:57.647531 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:57.647536 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:57.647537 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:57.647762 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:57.647953 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:57.647953 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:57.647953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:57.648048 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:57.648257 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:57.648261 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:57.648371 (+ 110us) client_negotiation.cc:769] Sending connection context
0926 05:41:57.648451 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:41:57.648565 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":104,"threads_started":1}
I20260926 05:41:57.649366 30445 heartbeater.cc:347] Connected to a master server at 127.23.213.254:39173
I20260926 05:41:57.649463 30445 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:57.649668 30445 heartbeater.cc:511] Master 127.23.213.254:39173 requested a full tablet report, sending...
I20260926 05:41:57.650094 29986 ts_manager.cc:194] Registered new tserver with Master: c0e8f6d78dc944d4b9311ef53c06817d (127.23.213.195:34517)
I20260926 05:41:57.650457 29986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:52749
I20260926 05:41:57.655599 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/test-admin.keytab.
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.670187 24407 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 05:41:57.672049 30452 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:57.671092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38768 (local address 127.23.213.193:39899)
0926 05:41:57.671285 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.671290 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.671301 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:41:57.671329 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.671331 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.671345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.671427 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.671965 (+ 538us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38768: BlockingRecv error: recv got EOF from 127.0.0.1:38768 (error 108)
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":90,"threads_started":1}
W20260926 05:41:57.672052 24407 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc)
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.673034 24407 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.673967 24407 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:57.683379 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.674592 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38774 (local address 127.23.213.193:39899)
0926 05:41:57.674648 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.674650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.674974 (+ 324us) server_negotiation.cc:407] Connection header received
0926 05:41:57.675051 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.675053 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.675068 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.675145 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.675556 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.676760 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.677542 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.678019 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.678067 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.680429 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:57.680437 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:57.680438 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:57.680445 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:57.682176 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.682542 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.682546 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.682547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.682608 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.682815 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.682816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.682816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.682958 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:57.683105 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:57.683184 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 05:41:57.683261 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
W20260926 05:41:57.685313 30452 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:57.684653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.23.213.193:39899)
0926 05:41:57.684693 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.684696 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.684735 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:41:57.684808 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.684809 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.684819 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.684872 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.685264 (+ 392us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38786: BlockingRecv error: recv got EOF from 127.0.0.1:38786 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:41:57.685395 24407 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc' of type FILE
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.686496 24407 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.687362 24407 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:57.694172 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.687938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38802 (local address 127.23.213.193:39899)
0926 05:41:57.687993 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.687995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.688048 (+ 53us) server_negotiation.cc:407] Connection header received
0926 05:41:57.688099 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.688100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.688108 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.688151 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.688500 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.688961 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.689613 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.689852 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.689901 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.691905 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:57.691906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:57.691906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:57.691912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:57.693040 (+ 1128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.693443 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.693444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.693444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.693492 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.693664 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.693665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.693665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.693802 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:57.693869 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:57.694016 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 05:41:57.694083 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 05:41:57.695705 30452 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:41:57.695187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38808 (local address 127.23.213.193:39899)
0926 05:41:57.695234 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.695236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.695333 (+ 97us) server_negotiation.cc:407] Connection header received
0926 05:41:57.695380 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.695381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.695388 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.695429 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.695667 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38808: BlockingRecv error: recv got EOF from 127.0.0.1:38808 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260926 05:41:57.695783 24407 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790401292175461-24407-0/krb5kdc/krb5cc' of type FILE
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.696746 24407 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:57.697564 24407 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[29941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401317, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:41:57.703836 30452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:57.698136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38816 (local address 127.23.213.193:39899)
0926 05:41:57.698201 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:57.698202 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:57.698210 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:41:57.698257 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:57.698258 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:57.698265 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:57.698307 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:57.698610 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.699050 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.699697 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:57.699865 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:57.699911 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:57.701854 (+ 1943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:57.701855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:57.701855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:57.701861 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:57.702968 (+ 1107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.703253 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.703253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.703254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.703298 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:57.703438 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:57.703439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:57.703439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:57.703511 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:57.703585 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:57.703667 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 05:41:57.703719 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 05:41:57.704483 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30048
I20260926 05:41:57.709486 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30182
I20260926 05:41:57.714375 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30316
I20260926 05:41:57.718706 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 29954
2026-09-26T05:41:57Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1889 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[30465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[30465](info): set up 2 sockets
Sep 26 05:41:57 dist-test-slave-1fdt krb5kdc[30465](info): commencing operation
krb5kdc: starting...
W20260926 05:41:58.766086 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:41:58 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:41:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:41:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab.
Entry for principal oryx/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab.
I20260926 05:41:58.888176 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:35903
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab
--principal=oryx/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:35903
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:58.970044 30478 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:58.970299 30478 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:58.970340 30478 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:58.971501 30478 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:58.971570 30478 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:58.971593 30478 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:58.971613 30478 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:58.971642 30478 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:58.973109 30478 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:35903
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab
--principal=oryx/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:35903
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30478
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:58.973400 30478 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:58.973644 30478 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:58.978498 30484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:58.978598 30478 server_base.cc:1034] running on GCE node
W20260926 05:41:58.978601 30486 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:58.978806 30483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:58.979043 30478 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:58.979338 30478 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:58.980501 30478 hybrid_clock.cc:648] HybridClock initialized: now 1790401318980491 us; error 29 us; skew 500 ppm
Sep 26 05:41:58 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401318, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:58.982941 30478 init.cc:377] Logged in from keytab as oryx/127.23.213.254@KRBTEST.COM (short username oryx)
I20260926 05:41:58.983372 30478 webserver.cc:460] Webserver started at http://127.23.213.254:38773/ using document root <none> and password file <none>
I20260926 05:41:58.983618 30478 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:58.983685 30478 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:58.983829 30478 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:58.984863 30478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "dccae96291a9413783a14543383a29ca"
format_stamp: "Formatted at 2026-09-26 05:41:58 on dist-test-slave-1fdt"
server_key: "8dc16d8df28971eb7faa892b073ebdcd"
server_key_iv: "ce28a2ec3a660ed16e351df5354efcd7"
server_key_version: "encryptionkey@0"
I20260926 05:41:58.985234 30478 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "dccae96291a9413783a14543383a29ca"
format_stamp: "Formatted at 2026-09-26 05:41:58 on dist-test-slave-1fdt"
server_key: "8dc16d8df28971eb7faa892b073ebdcd"
server_key_iv: "ce28a2ec3a660ed16e351df5354efcd7"
server_key_version: "encryptionkey@0"
I20260926 05:41:58.986608 30478 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:58.987406 30493 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:58.987586 30478 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:58.987689 30478 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "dccae96291a9413783a14543383a29ca"
format_stamp: "Formatted at 2026-09-26 05:41:58 on dist-test-slave-1fdt"
server_key: "8dc16d8df28971eb7faa892b073ebdcd"
server_key_iv: "ce28a2ec3a660ed16e351df5354efcd7"
server_key_version: "encryptionkey@0"
I20260926 05:41:58.987761 30478 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:59.001108 30478 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:59.003329 30478 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:59.003487 30478 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:59.007449 30478 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:35903
I20260926 05:41:59.007503 30545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:35903 every 8 connection(s)
I20260926 05:41:59.007871 30478 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:41:59.008872 30546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:59.011152 30546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: Bootstrap starting.
I20260926 05:41:59.011727 30546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:59.012013 30546 log.cc:824] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:59.012740 30546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: No bootstrap required, opened a new log
I20260926 05:41:59.012913 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30478
I20260926 05:41:59.013036 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:41:59.013219 24407 external_mini_cluster.cc:1428] Setting key a7eb47a7d8a35bc15580a3012d1497e7
I20260926 05:41:59.014791 30546 raft_consensus.cc:348] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } }
I20260926 05:41:59.014958 30546 raft_consensus.cc:374] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:59.014998 30546 raft_consensus.cc:729] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dccae96291a9413783a14543383a29ca, State: Initialized, Role: FOLLOWER
I20260926 05:41:59.015159 30546 consensus_queue.cc:260] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } }
I20260926 05:41:59.015254 30546 raft_consensus.cc:388] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:41:59.015288 30546 raft_consensus.cc:482] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:41:59.015337 30546 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:59.015995 30546 raft_consensus.cc:504] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } }
I20260926 05:41:59.016080 30546 leader_election.cc:302] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dccae96291a9413783a14543383a29ca; no voters:
I20260926 05:41:59.016252 30546 leader_election.cc:288] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:41:59.016290 30551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:41:59.016404 30551 raft_consensus.cc:686] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 1 LEADER]: Becoming Leader. State: Replica: dccae96291a9413783a14543383a29ca, State: Running, Role: LEADER
I20260926 05:41:59.016525 30546 sys_catalog.cc:564] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:59.016480 30551 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } }
I20260926 05:41:59.016827 30552 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } }
I20260926 05:41:59.016834 30553 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccae96291a9413783a14543383a29ca. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } }
I20260926 05:41:59.016996 30553 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:59.017194 30552 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:59.017359 30558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:41:59.018151 30558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:41:59.020457 30558 catalog_manager.cc:1347] Generated new cluster ID: b9443604a1104cce9a2871a783d8a847
I20260926 05:41:59.020516 30558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:41:59.025238 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.013729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57176 (local address 127.23.213.254:35903)
0926 05:41:59.013912 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.013920 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.013944 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:41:59.014773 (+ 829us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.014779 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.014824 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.014935 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.015727 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.016924 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.018036 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.018809 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.018944 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.021480 (+ 2536us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.021484 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.021484 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.021496 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.023564 (+ 2068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.023941 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.023944 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.023945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.024074 (+ 129us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.024274 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.024274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.024275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.024635 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.024834 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.024899 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.025035 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260926 05:41:59.031302 30558 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:41:59.031939 30558 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:41:59.040031 30558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: Generated new TSK 0
I20260926 05:41:59.040210 30558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab.
Entry for principal oryx/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab.
I20260926 05:41:59.066648 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35903
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab
--principal=oryx/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:59.154217 30572 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:59.154425 30572 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:59.154448 30572 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:59.155602 30572 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:59.155646 30572 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:59.155704 30572 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:41:59.157176 30572 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab
--principal=oryx/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30572
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:59.157430 30572 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:59.157672 30572 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:59.162721 30580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.162734 30577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.162732 30578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:59.162819 30572 server_base.cc:1034] running on GCE node
I20260926 05:41:59.163096 30572 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:59.163336 30572 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:59.164513 30572 hybrid_clock.cc:648] HybridClock initialized: now 1790401319164495 us; error 46 us; skew 500 ppm
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:59.167039 30572 init.cc:377] Logged in from keytab as oryx/127.23.213.193@KRBTEST.COM (short username oryx)
I20260926 05:41:59.167456 30572 webserver.cc:460] Webserver started at http://127.23.213.193:35531/ using document root <none> and password file <none>
I20260926 05:41:59.167709 30572 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:59.167778 30572 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:59.167910 30572 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:59.168936 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "e2add9c82ce14ffeb3db41d11218936a"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "a4ef7babd7027d79a2abf01d62e5a9d5"
server_key_iv: "adf16d7c8bbb9e672c34b06ec390009d"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.169301 30572 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "e2add9c82ce14ffeb3db41d11218936a"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "a4ef7babd7027d79a2abf01d62e5a9d5"
server_key_iv: "adf16d7c8bbb9e672c34b06ec390009d"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.170681 30572 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:41:59.171401 30587 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.171569 30572 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:41:59.171679 30572 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "e2add9c82ce14ffeb3db41d11218936a"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "a4ef7babd7027d79a2abf01d62e5a9d5"
server_key_iv: "adf16d7c8bbb9e672c34b06ec390009d"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.171763 30572 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:59.178499 30572 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:59.180248 30572 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:59.180402 30572 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:59.180614 30572 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:59.180918 30572 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:59.180979 30572 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.181010 30572 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:59.181036 30572 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.186636 30572 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:33691
I20260926 05:41:59.186730 30700 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:33691 every 8 connection(s)
I20260926 05:41:59.187018 30572 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:41:59.191905 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30572
I20260926 05:41:59.192037 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:41:59.192219 24407 external_mini_cluster.cc:1428] Setting key 8ec55181fd2857538881da3748cf83ff
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:41:59.196934 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.187722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:49193 (local address 127.23.213.254:35903)
0926 05:41:59.187823 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.187825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.187995 (+ 170us) server_negotiation.cc:407] Connection header received
0926 05:41:59.188444 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.188444 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.188454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.188525 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.189442 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.190014 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.191041 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.191244 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.191301 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.193694 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.193696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.193696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.193704 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.195427 (+ 1723us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.195938 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.195938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.195939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.195989 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.196274 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.196275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.196275 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.196378 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.196434 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.196777 (+ 343us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.196844 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260926 05:41:59.197191 30703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.187689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:49193)
0926 05:41:59.187875 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.187909 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.188366 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.188611 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.188614 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.189176 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.189362 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.189366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.190090 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.190091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.190972 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.190974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.191319 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.191319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.191415 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.191974 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.191982 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.193593 (+ 1611us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.195507 (+ 1914us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.195514 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.195516 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.195842 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.196053 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.196054 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.196055 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.196208 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.196448 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.196455 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.196632 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.196757 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.196931 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":113,"threads_started":1}
I20260926 05:41:59.197788 30701 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35903
I20260926 05:41:59.197890 30701 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:59.198105 30701 heartbeater.cc:511] Master 127.23.213.254:35903 requested a full tablet report, sending...
I20260926 05:41:59.198637 30510 ts_manager.cc:194] Registered new tserver with Master: e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691)
I20260926 05:41:59.199247 30510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:49193
WARNING: no policy specified for oryx/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab.
Entry for principal oryx/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab.
I20260926 05:41:59.216725 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35903
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab
--principal=oryx/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:59.296278 30706 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:59.296481 30706 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:59.296504 30706 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:59.297585 30706 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:59.297616 30706 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:59.297641 30706 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:41:59.299008 30706 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab
--principal=oryx/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30706
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:59.299190 30706 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:59.299357 30706 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:59.304152 30712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.304138 30711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.304129 30714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:59.304466 30706 server_base.cc:1034] running on GCE node
I20260926 05:41:59.304668 30706 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:59.304905 30706 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:59.306077 30706 hybrid_clock.cc:648] HybridClock initialized: now 1790401319306056 us; error 33 us; skew 500 ppm
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:59.308429 30706 init.cc:377] Logged in from keytab as oryx/127.23.213.194@KRBTEST.COM (short username oryx)
I20260926 05:41:59.308849 30706 webserver.cc:460] Webserver started at http://127.23.213.194:42037/ using document root <none> and password file <none>
I20260926 05:41:59.309079 30706 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:59.309144 30706 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:59.309288 30706 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:59.310395 30706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "327e803d0d744eb8933eff3fc78e4d2d"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "ec38b67031f397320ce637d09e2f4afe"
server_key_iv: "c678796554833661d457d89a1146a673"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.310743 30706 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "327e803d0d744eb8933eff3fc78e4d2d"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "ec38b67031f397320ce637d09e2f4afe"
server_key_iv: "c678796554833661d457d89a1146a673"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.312064 30706 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:59.312780 30721 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.313043 30706 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:59.313153 30706 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "327e803d0d744eb8933eff3fc78e4d2d"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "ec38b67031f397320ce637d09e2f4afe"
server_key_iv: "c678796554833661d457d89a1146a673"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.313227 30706 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:59.321321 30706 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:59.323331 30706 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:59.323485 30706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:59.323709 30706 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:59.324025 30706 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:59.324081 30706 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.324121 30706 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:59.324149 30706 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.329886 30706 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:36013
I20260926 05:41:59.329928 30834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:36013 every 8 connection(s)
I20260926 05:41:59.330251 30706 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:41:59.331411 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30706
I20260926 05:41:59.331517 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:41:59.331722 24407 external_mini_cluster.cc:1428] Setting key c6129c5a1bd9bd1826cc1dfab40560d4
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
WARNING: no policy specified for oryx/127.23.213.195@KRBTEST.COM; defaulting to no policy
I20260926 05:41:59.340508 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.330948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:60655 (local address 127.23.213.254:35903)
0926 05:41:59.331013 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.331030 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.331211 (+ 181us) server_negotiation.cc:407] Connection header received
0926 05:41:59.331974 (+ 763us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.331976 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.331989 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.332047 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.333320 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.333913 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.334924 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.335152 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.335227 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.337383 (+ 2156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.337385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.337385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.337393 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.339090 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.339474 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.339476 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.339476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.339537 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.339875 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.339876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.339877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.340047 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.340123 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.340377 (+ 254us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.340444 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:41:59.340699 30837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.330919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:60655)
0926 05:41:59.331086 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.331125 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.331860 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.332113 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.332117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.332925 (+ 808us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.333221 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.333227 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.334004 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.334005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.334828 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.334830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.335246 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.335247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.335384 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.335944 (+ 560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.335953 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.337291 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.339173 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.339178 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.339180 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.339394 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.339653 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.339654 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.339654 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.339796 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.340131 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.340136 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.340253 (+ 117us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.340340 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.340457 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":82,"threads_started":1}
I20260926 05:41:59.341288 30835 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35903
I20260926 05:41:59.341369 30835 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:59.341506 30835 heartbeater.cc:511] Master 127.23.213.254:35903 requested a full tablet report, sending...
I20260926 05:41:59.341920 30510 ts_manager.cc:194] Registered new tserver with Master: 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194:36013)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.195@KRBTEST.COM" created.
I20260926 05:41:59.342391 30510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:60655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab.
Entry for principal oryx/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab.
I20260926 05:41:59.357084 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35903
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab
--principal=oryx/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:59.439394 30840 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:59.439610 30840 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:59.439634 30840 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:59.440793 30840 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:59.440838 30840 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:59.440896 30840 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:41:59.442431 30840 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab
--principal=oryx/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35903
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30840
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:41:59.442631 30840 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:59.442885 30840 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:59.447896 30845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.447911 30846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.448096 30848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:59.448060 30840 server_base.cc:1034] running on GCE node
I20260926 05:41:59.448305 30840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:59.448510 30840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:59.449646 30840 hybrid_clock.cc:648] HybridClock initialized: now 1790401319449629 us; error 32 us; skew 500 ppm
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:59.452100 30840 init.cc:377] Logged in from keytab as oryx/127.23.213.195@KRBTEST.COM (short username oryx)
I20260926 05:41:59.452526 30840 webserver.cc:460] Webserver started at http://127.23.213.195:42363/ using document root <none> and password file <none>
I20260926 05:41:59.452775 30840 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:59.452847 30840 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:59.453001 30840 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:59.454147 30840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "31bf19fb701148f58f69c8b548f2cbc5"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "2d60e63e3b4d1063236babb91e4f9f83"
server_key_iv: "647ddd049c6b7a6fdc8838dc51091af3"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.454591 30840 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "31bf19fb701148f58f69c8b548f2cbc5"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "2d60e63e3b4d1063236babb91e4f9f83"
server_key_iv: "647ddd049c6b7a6fdc8838dc51091af3"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.455922 30840 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:41:59.456641 30855 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.456802 30840 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:41:59.456902 30840 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "31bf19fb701148f58f69c8b548f2cbc5"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "2d60e63e3b4d1063236babb91e4f9f83"
server_key_iv: "647ddd049c6b7a6fdc8838dc51091af3"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.456980 30840 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:59.480258 30840 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:59.482141 30840 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:59.482307 30840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:59.482551 30840 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:41:59.482885 30840 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:41:59.482940 30840 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.482982 30840 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:41:59.483031 30840 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.488811 30840 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:38665
I20260926 05:41:59.488852 30968 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:38665 every 8 connection(s)
I20260926 05:41:59.489169 30840 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:41:59.492103 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30840
I20260926 05:41:59.492182 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:41:59.492416 24407 external_mini_cluster.cc:1428] Setting key 074acc1411673a49094181933465b5a9
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:41:59.498347 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.490008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:40789 (local address 127.23.213.254:35903)
0926 05:41:59.490089 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.490091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.490247 (+ 156us) server_negotiation.cc:407] Connection header received
0926 05:41:59.490717 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.490718 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.490728 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.490778 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.491741 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.492324 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.493672 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.493923 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.493989 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.495963 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.495964 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.495964 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.495971 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.497149 (+ 1178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.497511 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.497512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.497512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.497559 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.497821 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.497822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.497822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.497933 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.497986 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.498220 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.498282 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:41:59.498535 30971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.489967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:40789)
0926 05:41:59.490125 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.490162 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.490635 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.490891 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.490895 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.491469 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.491666 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.491671 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.492409 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.492411 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.493601 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.493604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.494001 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.494002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.494116 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.494552 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.494559 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.495885 (+ 1326us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.497220 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.497226 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.497228 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.497449 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.497614 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.497614 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.497614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.497708 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.497990 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.497994 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.498109 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.498191 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.498306 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 05:41:59.499054 30969 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35903
I20260926 05:41:59.499145 30969 heartbeater.cc:464] Registering TS with master...
I20260926 05:41:59.499289 30969 heartbeater.cc:511] Master 127.23.213.254:35903 requested a full tablet report, sending...
I20260926 05:41:59.499644 30510 ts_manager.cc:194] Registered new tserver with Master: 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195:38665)
I20260926 05:41:59.500007 30510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:40789
I20260926 05:41:59.504043 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.213.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.253.keytab.
Entry for principal oryx/127.23.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.253.keytab.
I20260926 05:41:59.529273 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:39357
--webserver_interface=127.23.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.253.keytab
--principal=oryx/127.23.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:35903,127.23.213.253:39357
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:41:59.610357 30974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:41:59.610555 30974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:41:59.610577 30974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:41:59.611634 30974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:41:59.611668 30974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:41:59.611682 30974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:41:59.611696 30974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:41:59.611708 30974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:41:59.613078 30974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:35903,127.23.213.253:39357
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.253.keytab
--principal=oryx/127.23.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:39357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
W20260926 05:41:59.613171 30974 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.23.213.254:35903,127.23.213.253:39357'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 05:41:59.613277 30974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:41:59.613504 30974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:59.618438 30982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:59.618570 30974 server_base.cc:1034] running on GCE node
W20260926 05:41:59.618481 30980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.618491 30979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:59.618896 30974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:59.619218 30974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:59.620397 30974 hybrid_clock.cc:648] HybridClock initialized: now 1790401319620376 us; error 38 us; skew 500 ppm
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:59.622992 30974 init.cc:377] Logged in from keytab as oryx/127.23.213.253@KRBTEST.COM (short username oryx)
I20260926 05:41:59.623380 30974 webserver.cc:460] Webserver started at http://127.23.213.253:34849/ using document root <none> and password file <none>
I20260926 05:41:59.623624 30974 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:59.623672 30974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:59.623800 30974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:41:59.624874 30974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/instance:
uuid: "4fa2426091a94d768868fdf7ed313b13"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "f92ed408edfc5f61d4e861bb043a510e"
server_key_iv: "0de717b693bc1841f6c3cec117d08fee"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.625234 30974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal/instance:
uuid: "4fa2426091a94d768868fdf7ed313b13"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "f92ed408edfc5f61d4e861bb043a510e"
server_key_iv: "0de717b693bc1841f6c3cec117d08fee"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.626662 30974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:41:59.627452 30989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.627657 30974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:59.627746 30974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
uuid: "4fa2426091a94d768868fdf7ed313b13"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "f92ed408edfc5f61d4e861bb043a510e"
server_key_iv: "0de717b693bc1841f6c3cec117d08fee"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.627818 30974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:59.637187 30974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:41:59.639359 30974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:41:59.639509 30974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:41:59.643376 30974 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.253:39357
I20260926 05:41:59.643445 31041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.253:39357 every 8 connection(s)
I20260926 05:41:59.643872 30974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
I20260926 05:41:59.645242 31042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:41:59.647953 31042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.253@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:41:59.654155 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 30974
I20260926 05:41:59.657663 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.648885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57184 (local address 127.23.213.254:35903)
0926 05:41:59.648965 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.648968 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.649208 (+ 240us) server_negotiation.cc:407] Connection header received
0926 05:41:59.649662 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.649663 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.649678 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.649818 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.650472 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.650975 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.652026 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.652220 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.652303 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.654677 (+ 2374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.654678 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.654679 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.654686 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.656269 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.656658 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.656659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.656659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.656720 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.656980 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.656981 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.656981 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.657135 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.657232 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.657493 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.657570 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:41:59.657917 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.648874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57184)
0926 05:41:59.649072 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.649114 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.649555 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.649903 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.649907 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.650165 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.650363 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.650368 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.651094 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.651095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.651922 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.651924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.652313 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.652314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.652443 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.652963 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.652971 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.654514 (+ 1543us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.656360 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.656366 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.656368 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.656581 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.656787 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.656788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.656788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.656898 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.657225 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.657231 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.657362 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.657473 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.657630 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":121,"threads_started":1}
I20260926 05:41:59.658550 31042 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.213.253@KRBTEST.COM
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.253@KRBTEST.COM for oryx/127.23.213.253@KRBTEST.COM
I20260926 05:41:59.664258 31046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.654662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42022 (local address 127.23.213.253:39357)
0926 05:41:59.654972 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.654976 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.654999 (+ 23us) server_negotiation.cc:407] Connection header received
0926 05:41:59.655023 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.655024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.655038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.655128 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.655968 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.656623 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.657413 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.657783 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.657887 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.660711 (+ 2824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.660718 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.660719 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.660725 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.662564 (+ 1839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.663057 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.663060 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.663061 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.663141 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.663397 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.663399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.663399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.663647 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.663876 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.663983 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.664063 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":78,"threads_started":1}
I20260926 05:41:59.666991 31044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.658818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.0.0.1:42030)
0926 05:41:59.658859 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.658871 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.658919 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.659144 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.659145 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.659286 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.659326 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.659327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.660292 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.660293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.661025 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.661028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.661462 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.661463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.661558 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.661892 (+ 334us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.661905 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.663549 (+ 1644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.666157 (+ 2608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.666158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.666158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.666354 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.666560 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.666560 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.666561 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.666608 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.666786 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.666786 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.666816 (+ 30us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.666861 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.666929 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260926 05:41:59.667124 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.658901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.23.213.253:39357)
0926 05:41:59.659051 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.659052 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.659057 (+ 5us) server_negotiation.cc:407] Connection header received
0926 05:41:59.659068 (+ 11us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.659069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.659074 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.659142 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.659413 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.660191 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.661100 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.661375 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.661442 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.663650 (+ 2208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.663651 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.663652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.663658 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.666077 (+ 2419us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.666443 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.666443 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.666444 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.666499 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.666645 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.666646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.666646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.666738 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.666793 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.666878 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.666984 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":136,"thread_start_us":115,"threads_started":1}
I20260926 05:41:59.668569 31042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Bootstrap starting.
I20260926 05:41:59.669202 31042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Neither blocks nor log segments found. Creating new log.
I20260926 05:41:59.669482 31042 log.cc:824] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Log is configured to *not* fsync() on all Append() calls
I20260926 05:41:59.670346 31042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: No bootstrap required, opened a new log
I20260926 05:41:59.671749 31042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } }
I20260926 05:41:59.671873 31042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:41:59.671896 31042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4fa2426091a94d768868fdf7ed313b13, State: Initialized, Role: FOLLOWER
I20260926 05:41:59.672061 31042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } }
I20260926 05:41:59.672317 31049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } } }
I20260926 05:41:59.672372 31042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:41:59.672438 31049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:41:59.674283 31060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:41:59.674340 31060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:41:59.675834 30974 master_runner.cc:428] Detected that this master 4fa2426091a94d768868fdf7ed313b13 is joining an existing cluster
I20260926 05:41:59.675894 30974 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.213.253:39357
I20260926 05:41:59.683501 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.676805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.23.213.253:39357)
0926 05:41:59.676871 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.676873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.677003 (+ 130us) server_negotiation.cc:407] Connection header received
0926 05:41:59.677126 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.677127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.677136 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.677193 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.677603 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.678179 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.678927 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.679154 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.679244 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.680429 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.680430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.680430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.680436 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.682136 (+ 1700us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.682596 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.682597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.682598 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.682669 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.682870 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.682870 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.682871 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.683000 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.683052 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.683267 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.683357 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:41:59.683847 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.677016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57196 (local address 127.23.213.254:35903)
0926 05:41:59.677062 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.677063 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.677149 (+ 86us) server_negotiation.cc:407] Connection header received
0926 05:41:59.677261 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.677262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.677273 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.677327 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.677721 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.678530 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.679619 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.679914 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.679955 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.680935 (+ 980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.680937 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.680937 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.680944 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.682610 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.682988 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.682989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.682989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.683046 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.683269 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.683270 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.683271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.683395 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.683439 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.683675 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.683757 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:41:59.683925 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.676725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.0.0.1:42038)
0926 05:41:59.676926 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.676950 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.677027 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.677197 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.677198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.677463 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.677520 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.677522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.678276 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.678277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.678830 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.678832 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.679244 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.679244 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.679310 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.679794 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.679800 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.680121 (+ 321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.682264 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.682265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.682266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.682514 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.682737 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.682738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.682738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.682807 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.683092 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.683093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.683131 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.683656 (+ 525us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.683763 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":3,"thread_start_us":91,"threads_started":1}
I20260926 05:41:59.684260 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.676971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57196)
0926 05:41:59.677088 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.677097 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.677177 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.677331 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.677332 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.677549 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.677621 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.677623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.678639 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.678640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.679516 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.679519 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.680030 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.680031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.680103 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.680441 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.680446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.680833 (+ 387us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.682700 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.682700 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.682701 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.682902 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.683138 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.683139 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.683139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.683198 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.683486 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.683487 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.683535 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.683959 (+ 424us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.684159 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":1,"thread_start_us":59,"threads_started":1}
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401318, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.254@KRBTEST.COM for oryx/127.23.213.253@KRBTEST.COM
I20260926 05:41:59.694499 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.686763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42052 (local address 127.23.213.253:39357)
0926 05:41:59.686803 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.686805 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.686873 (+ 68us) server_negotiation.cc:407] Connection header received
0926 05:41:59.686966 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.686967 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.686977 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.687041 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.687531 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.688203 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.689344 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.689636 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.689676 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.692063 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.692065 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.692065 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.692071 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.693311 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.693799 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.693801 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.693801 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.693856 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.694062 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.694063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.694063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.694158 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.694207 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.694386 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.694440 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:41:59.694757 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.686556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.0.0.1:42052)
0926 05:41:59.686708 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.686821 (+ 113us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.686897 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.687036 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.687039 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.687381 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.687448 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.687451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.688289 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.688290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.689243 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.689246 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.689785 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.689786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.689865 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.690340 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.690345 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.691950 (+ 1605us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.693391 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.693398 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.693400 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.693663 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.693920 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.693921 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.693922 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.693988 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.694218 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.694224 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.694278 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.694371 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.694490 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260926 05:41:59.695324 30510 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.213.253:39357: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4fa2426091a94d768868fdf7ed313b13"
member_type: NON_VOTER
last_known_addr {
host: "127.23.213.253"
port: 39357
}
attrs {
promote: true
}
}
dest_uuid: "dccae96291a9413783a14543383a29ca"
cas_config_opid_index: -1
I20260926 05:41:59.695650 30510 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } }
I20260926 05:41:59.696165 31070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } }
I20260926 05:41:59.696449 31016 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:41:59.742199 31016 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Refusing update from remote peer dccae96291a9413783a14543383a29ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 05:41:59.742267 31070 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.213.253:39357
I20260926 05:41:59.742568 30974 master.cc:537] Master@127.23.213.253:39357 shutting down...
I20260926 05:41:59.744043 31071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:41:59.745039 30494 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260926 05:41:59.745630 30494 consensus_peers.cc:487] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca -> Peer 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253:39357): Couldn't send request to peer 4fa2426091a94d768868fdf7ed313b13. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 05:41:59.746915 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } }
I20260926 05:41:59.747006 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:41:59.748027 30974 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:41:59.748134 30974 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:41:59.748193 30974 tablet_replica.cc:330] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: stopping tablet replica
WARNING: no policy specified for oryx/127.23.213.252@KRBTEST.COM; defaulting to no policy
I20260926 05:41:59.761600 30974 master.cc:552] Master@127.23.213.253:39357 shutdown complete.
I20260926 05:41:59.761662 30974 master_runner.cc:305] Clearing existing system tablet
I20260926 05:41:59.761833 30974 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:59.761889 30974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:59.762672 30974 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:59.763240 31076 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.763427 30974 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:41:59.763495 30974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:41:59.763522 30974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
uuid: "4fa2426091a94d768868fdf7ed313b13"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "f92ed408edfc5f61d4e861bb043a510e"
server_key_iv: "0de717b693bc1841f6c3cec117d08fee"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.764384 30974 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:41:59.842227 30974 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 05:41:59.842319 30974 log.cc:1196] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 05:41:59.842547 30974 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Deleting consensus metadata
I20260926 05:41:59.869560 30974 master_runner.cc:315] Copying system tablet from 127.23.213.254:35903
I20260926 05:41:59.870316 30974 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: tablet copy: Beginning tablet copy session from remote peer at address 127.23.213.254:35903
I20260926 05:41:59.877038 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.870759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57204 (local address 127.23.213.254:35903)
0926 05:41:59.870821 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:41:59.870823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:41:59.870942 (+ 119us) server_negotiation.cc:407] Connection header received
0926 05:41:59.871016 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:41:59.871017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:41:59.871028 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:41:59.871087 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:41:59.871483 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.872000 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.872815 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.873013 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.873135 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.874171 (+ 1036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:41:59.874172 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:41:59.874173 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:41:59.874180 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:41:59.875874 (+ 1694us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.876221 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.876222 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.876222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.876321 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:41:59.876507 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:41:59.876508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:41:59.876509 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:41:59.876635 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:41:59.876711 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:41:59.876862 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 05:41:59.876941 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:41:59.877061 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:41:59.870698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57204)
0926 05:41:59.870858 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:41:59.870869 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:41:59.870936 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:41:59.871107 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:41:59.871108 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:41:59.871348 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:41:59.871406 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.871408 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.872085 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.872086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.872725 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:41:59.872728 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:41:59.873131 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:41:59.873132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:41:59.873196 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:41:59.873689 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:41:59.873694 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:41:59.874084 (+ 390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:41:59.875951 (+ 1867us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.875952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.875952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.876130 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.876373 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:41:59.876374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:41:59.876374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:41:59.876435 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:41:59.876701 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:41:59.876702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:41:59.876749 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 05:41:59.876837 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:41:59.876967 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":114,"threads_started":1}
I20260926 05:41:59.877521 30530 tablet_copy_service.cc:140] P dccae96291a9413783a14543383a29ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4fa2426091a94d768868fdf7ed313b13 ({username='oryx', principal='oryx/127.23.213.253@KRBTEST.COM'} at 127.0.0.1:57204)
I20260926 05:41:59.877735 30530 tablet_copy_service.cc:161] P dccae96291a9413783a14543383a29ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4fa2426091a94d768868fdf7ed313b13 at {username='oryx', principal='oryx/127.23.213.253@KRBTEST.COM'} at 127.0.0.1:57204: session id = 4fa2426091a94d768868fdf7ed313b13-00000000000000000000000000000000
I20260926 05:41:59.878419 30530 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 05:41:59.879011 30974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 05:41:59.880060 30974 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: tablet copy: Starting download of 0 data blocks...
I20260926 05:41:59.880220 30974 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: tablet copy: Starting download of 1 WAL segments...
I20260926 05:41:59.881106 30974 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 05:41:59.937459 30530 tablet_copy_service.cc:342] P dccae96291a9413783a14543383a29ca: Request end of tablet copy session 4fa2426091a94d768868fdf7ed313b13-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.213.253@KRBTEST.COM'} at 127.0.0.1:57204
I20260926 05:41:59.937587 30530 tablet_copy_service.cc:433] P dccae96291a9413783a14543383a29ca: ending tablet copy session 4fa2426091a94d768868fdf7ed313b13-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4fa2426091a94d768868fdf7ed313b13
I20260926 05:41:59.940158 30974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:41:59.941411 31083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:59.941557 30974 server_base.cc:1034] running on GCE node
W20260926 05:41:59.941417 31084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:41:59.941417 31086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:41:59.941875 30974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:41:59.942044 30974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:41:59.943158 30974 hybrid_clock.cc:648] HybridClock initialized: now 1790401319943151 us; error 45 us; skew 500 ppm
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.252@KRBTEST.COM" created.
Sep 26 05:41:59 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401319, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:41:59.995638 30974 init.cc:377] Logged in from keytab as oryx/127.23.213.253@KRBTEST.COM (short username oryx)
I20260926 05:41:59.995976 30974 webserver.cc:460] Webserver started at http://127.23.213.253:37277/ using document root <none> and password file <none>
I20260926 05:41:59.996136 30974 fs_manager.cc:359] Metadata directory not provided
I20260926 05:41:59.996188 30974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:41:59.997037 30974 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:59.997812 31094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:41:59.997990 30974 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:41:59.998050 30974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
uuid: "4fa2426091a94d768868fdf7ed313b13"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "f92ed408edfc5f61d4e861bb043a510e"
server_key_iv: "0de717b693bc1841f6c3cec117d08fee"
server_key_version: "encryptionkey@0"
I20260926 05:41:59.998126 30974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:00.008678 30974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:00.010458 30974 kserver.cc:163] Server-wide thread pool size limit: 3276
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.252.keytab.
Entry for principal oryx/127.23.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.252.keytab.
W20260926 05:42:00.012295 24407 mini_kdc.cc:262] Time spent creating service keytab for oryx/127.23.213.252: real 0.268s user 0.000s sys 0.003s
I20260926 05:42:00.012671 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:40177
--webserver_interface=127.23.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.252.keytab
--principal=oryx/127.23.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:42:00.013620 30974 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.253:39357
I20260926 05:42:00.014050 30974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
I20260926 05:42:00.014999 31146 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.253:39357 every 8 connection(s)
I20260926 05:42:00.015493 31148 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:42:00.015921 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Bootstrap starting.
I20260926 05:42:00.018540 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:42:00.018755 31148 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: Bootstrap complete.
I20260926 05:42:00.018956 31148 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } }
I20260926 05:42:00.019043 31148 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4fa2426091a94d768868fdf7ed313b13, State: Initialized, Role: LEARNER
I20260926 05:42:00.019101 31148 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } }
I20260926 05:42:00.019318 31148 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:00.019300 31151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } }
I20260926 05:42:00.019459 31151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:42:00.020740 31162 catalog_manager.cc:1259] Loaded cluster ID: b9443604a1104cce9a2871a783d8a847
I20260926 05:42:00.020828 31162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: loading cluster ID for follower catalog manager: success
I20260926 05:42:00.021862 31162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: acquiring CA information for follower catalog manager: success
I20260926 05:42:00.022131 31162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 05:42:00.101670 31147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:00.101955 31147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:00.101981 31147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:00.103101 31147 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:00.103140 31147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:00.103156 31147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:00.103171 31147 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:00.103185 31147 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:00.104707 31147 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.252.keytab
--principal=oryx/127.23.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:40177
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31147
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:00.104920 31147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:00.105103 31147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:00.109977 31168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.109961 31167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:00.110008 31147 server_base.cc:1034] running on GCE node
W20260926 05:42:00.109963 31170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:00.110457 31147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:00.110699 31147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:00.111862 31147 hybrid_clock.cc:648] HybridClock initialized: now 1790401320111855 us; error 33 us; skew 500 ppm
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:00.114427 31147 init.cc:377] Logged in from keytab as oryx/127.23.213.252@KRBTEST.COM (short username oryx)
I20260926 05:42:00.114912 31147 webserver.cc:460] Webserver started at http://127.23.213.252:39727/ using document root <none> and password file <none>
I20260926 05:42:00.115110 31147 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:00.115154 31147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:00.115249 31147 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:00.116452 31147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/instance:
uuid: "b3f821498f104da68674c7532371e0b4"
format_stamp: "Formatted at 2026-09-26 05:42:00 on dist-test-slave-1fdt"
server_key: "cb8ea994a8e025806614bcb585bb78d1"
server_key_iv: "4dcaafadaf14b3be7455be9411b93835"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.116911 31147 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal/instance:
uuid: "b3f821498f104da68674c7532371e0b4"
format_stamp: "Formatted at 2026-09-26 05:42:00 on dist-test-slave-1fdt"
server_key: "cb8ea994a8e025806614bcb585bb78d1"
server_key_iv: "4dcaafadaf14b3be7455be9411b93835"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.118422 31147 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:42:00.119280 31177 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.119562 31147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:00.119647 31147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
uuid: "b3f821498f104da68674c7532371e0b4"
format_stamp: "Formatted at 2026-09-26 05:42:00 on dist-test-slave-1fdt"
server_key: "cb8ea994a8e025806614bcb585bb78d1"
server_key_iv: "4dcaafadaf14b3be7455be9411b93835"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.119745 31147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:00.134312 31147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:00.136163 31147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:00.136317 31147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:00.140470 31147 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.252:40177
I20260926 05:42:00.140504 31229 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.252:40177 every 8 connection(s)
I20260926 05:42:00.140863 31147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
I20260926 05:42:00.141891 31230 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:00.143323 31230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.252@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:00.149631 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 31147
I20260926 05:42:00.150930 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.147603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.0.0.1:42056)
0926 05:42:00.147662 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.147673 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.147719 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.147988 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.147989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.147991 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:00.148040 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.148041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.148820 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.148821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.149824 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.149827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.150648 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.150649 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.150754 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.150783 (+ 29us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.150838 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.150876 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 05:42:00.151014 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.147689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42056 (local address 127.23.213.253:39357)
0926 05:42:00.147809 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.147810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.147818 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:00.147839 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.147839 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.147858 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.147918 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:00.148114 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.148732 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.149921 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.150497 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.150642 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.150760 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.150862 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.150922 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":56,"threads_started":1}
I20260926 05:42:00.151769 31121 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260926 05:42:00.151994 31151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccae96291a9413783a14543383a29ca. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } }
I20260926 05:42:00.152110 31151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:42:00.153085 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.144041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57216 (local address 127.23.213.254:35903)
0926 05:42:00.144106 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.144108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.144277 (+ 169us) server_negotiation.cc:407] Connection header received
0926 05:42:00.144730 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.144731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.144740 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.144799 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.145374 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.145914 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.146851 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.147033 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.147097 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.149407 (+ 2310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.149409 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.149409 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.149417 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.151711 (+ 2294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.152172 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.152173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.152174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.152241 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.152494 (+ 253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.152495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.152496 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.152610 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.152666 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.152965 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.153022 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:42:00.153326 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.143978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57216)
0926 05:42:00.144146 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.144187 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.144648 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.144875 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.144878 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.145118 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.145303 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.145306 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.145988 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.145988 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.146774 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.146777 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.147116 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.147117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.147255 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.147832 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.147840 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.149328 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.151782 (+ 2454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.151787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.151790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.152076 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.152308 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.152309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.152309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.152432 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.152659 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.152664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.152817 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.152951 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.153111 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":92,"threads_started":1}
I20260926 05:42:00.153930 31230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401318, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.213.252@KRBTEST.COM
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.252@KRBTEST.COM for oryx/127.23.213.253@KRBTEST.COM
I20260926 05:42:00.160456 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.150338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40236 (local address 127.23.213.252:40177)
0926 05:42:00.150509 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.150513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.150531 (+ 18us) server_negotiation.cc:407] Connection header received
0926 05:42:00.150552 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.150555 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.150568 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.150719 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.151448 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.152517 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.153561 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.153981 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.154060 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.156632 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.156639 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.156639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.156645 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.158621 (+ 1976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.159091 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.159094 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.159094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.159154 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.159337 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.159338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.159338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.159553 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.160177 (+ 624us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.160226 (+ 49us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.160329 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":76,"threads_started":1}
I20260926 05:42:00.161734 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.154200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.0.0.1:42058)
0926 05:42:00.154328 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.154338 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.154389 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.154509 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.154510 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.154661 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.154719 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.154720 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.155319 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.155320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.155933 (+ 613us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.155935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.156463 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.156464 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.156546 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.156941 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.156946 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.158434 (+ 1488us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.160549 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.160550 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.160550 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.160774 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.161033 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.161034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.161035 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.161096 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.161385 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.161385 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.161427 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.161515 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.161619 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20260926 05:42:00.161839 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.154276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42058 (local address 127.23.213.253:39357)
0926 05:42:00.154342 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.154343 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.154372 (+ 29us) server_negotiation.cc:407] Connection header received
0926 05:42:00.154432 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.154433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.154443 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.154504 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.154795 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.155227 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.156018 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.156369 (+ 351us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.156439 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.158544 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.158545 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.158545 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.158554 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.160448 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.160883 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.160885 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.160886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.160946 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.161167 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.161167 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.161168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.161309 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.161381 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.161564 (+ 183us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.161663 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:42:00.162182 31230 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.252@KRBTEST.COM for oryx/127.23.213.252@KRBTEST.COM
I20260926 05:42:00.168931 31232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.162462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.0.0.1:40244)
0926 05:42:00.162493 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.162504 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.162557 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.162682 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.162682 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.162814 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.162845 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.162847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.163611 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.163612 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.164334 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.164338 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.164793 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.164794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.164844 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.165232 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.165237 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.166463 (+ 1226us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.168005 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.168005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.168006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.168163 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.168363 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.168363 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.168364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.168414 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.168655 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.168656 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.168691 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.168757 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.168849 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260926 05:42:00.169067 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.162527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.23.213.252:40177)
0926 05:42:00.162566 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.162568 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.162575 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:42:00.162616 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.162617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.162626 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.162678 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.162919 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.163518 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.164415 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.164703 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.164793 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.166557 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.166558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.166559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.166572 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.167908 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.168243 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.168244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.168245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.168292 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.168483 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.168483 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.168484 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.168591 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.168662 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.168784 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.168939 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":84,"server-negotiator.queue_time_us":26}
I20260926 05:42:00.170487 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Bootstrap starting.
I20260926 05:42:00.171334 31230 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:00.171613 31230 log.cc:824] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:00.172382 31230 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: No bootstrap required, opened a new log
I20260926 05:42:00.173974 31230 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } }
I20260926 05:42:00.174098 31230 raft_consensus.cc:374] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:00.174145 31230 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3f821498f104da68674c7532371e0b4, State: Initialized, Role: FOLLOWER
I20260926 05:42:00.174278 31230 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } }
I20260926 05:42:00.174597 31236 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } } }
I20260926 05:42:00.174638 31230 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:00.174705 31236 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:42:00.176437 31247 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:42:00.176492 31247 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:42:00.178704 31147 master_runner.cc:428] Detected that this master b3f821498f104da68674c7532371e0b4 is joining an existing cluster
I20260926 05:42:00.178745 31147 master_runner.cc:432] Initiating AddMaster RPC to add 127.23.213.252:40177
I20260926 05:42:00.187197 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.180019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57218 (local address 127.23.213.254:35903)
0926 05:42:00.180060 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.180062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.180337 (+ 275us) server_negotiation.cc:407] Connection header received
0926 05:42:00.180465 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.180466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.180478 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.180507 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.180921 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.181688 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.182700 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.182974 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.183013 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.183988 (+ 975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.183989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.183990 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.183997 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.185734 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.186172 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.186173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.186174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.186241 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.186458 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.186459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.186460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.186627 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.186689 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.186992 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.187072 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:42:00.187537 31255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.180115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57218)
0926 05:42:00.180269 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.180279 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.180373 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.180566 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.180566 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.180774 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.180833 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.180835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.181801 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.181802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.182606 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.182608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.183094 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.183094 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.183163 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.183527 (+ 364us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.183531 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.183839 (+ 308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.185858 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.185859 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.185860 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.186084 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.186318 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.186319 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.186320 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.186385 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.186759 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.186760 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.186824 (+ 64us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.187293 (+ 469us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.187398 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
I20260926 05:42:00.187813 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.179841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42064 (local address 127.23.213.253:39357)
0926 05:42:00.179883 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.179885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.180298 (+ 413us) server_negotiation.cc:407] Connection header received
0926 05:42:00.180392 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.180393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.180403 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.180432 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.180858 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.181603 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.183196 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.183755 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.183801 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.184858 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.184859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.184860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.184868 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.186590 (+ 1722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.187017 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.187018 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.187019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.187073 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.187282 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.187283 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.187283 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.187405 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.187448 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.187656 (+ 208us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.187728 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:42:00.187826 31254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.179698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.0.0.1:42064)
0926 05:42:00.180239 (+ 541us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.180248 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.180322 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.180493 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.180494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.180703 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.180762 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.180764 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.181702 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.181703 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.183101 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.183104 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.183883 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.183884 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.183946 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.184359 (+ 413us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.184363 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.184745 (+ 382us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.186677 (+ 1932us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.186678 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.186678 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.186937 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.187156 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.187157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.187157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.187212 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.187495 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.187496 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.187531 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.187624 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.187723 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":64,"threads_started":1}
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401318, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.254@KRBTEST.COM for oryx/127.23.213.252@KRBTEST.COM
I20260926 05:42:00.200357 30701 heartbeater.cc:503] Master 127.23.213.254:35903 was elected leader, sending a full tablet report...
I20260926 05:42:00.205061 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.190592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.0.0.1:40260)
0926 05:42:00.190645 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.190655 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.190691 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.190963 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.190964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.191118 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.191151 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.191153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.191942 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.191943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.192438 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.192440 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.192888 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.192889 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.192957 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.193365 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.193370 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.194715 (+ 1345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.203896 (+ 9181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.203897 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.203898 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.204148 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.204374 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.204375 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.204375 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.204451 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.204713 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.204714 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.204766 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.204835 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.204956 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:42:00.205089 31234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.179855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40258 (local address 127.23.213.252:40177)
0926 05:42:00.179890 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.179892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.179900 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:00.179959 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.179960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.179969 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.180024 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.180416 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.181139 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.182167 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.182443 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.182476 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.202281 (+ 19805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.202282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.202282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.202290 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.203968 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.204324 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.204326 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.204326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.204384 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.204561 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.204563 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.204563 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.204695 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.204760 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.204907 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.205002 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":18}
I20260926 05:42:00.205283 31256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.190654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.23.213.252:40177)
0926 05:42:00.190836 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.190837 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.190843 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:42:00.190871 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.190872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.190880 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.190947 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.191238 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.191863 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.192541 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.192810 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.192888 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.194832 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.194834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.194834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.194841 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.203804 (+ 8963us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.204228 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.204229 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.204230 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.204282 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.204504 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.204504 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.204505 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.204638 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.204706 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.204861 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.205159 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":87,"threads_started":1}
I20260926 05:42:00.205626 31253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.179674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.0.0.1:40258)
0926 05:42:00.179820 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.179833 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.179897 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.180028 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.180029 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.180247 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.180324 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.180326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.181217 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.181218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.182089 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.182091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.182540 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.182541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.182593 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.183059 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.183064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.202196 (+ 19132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.204047 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.204048 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.204049 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.204247 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.204440 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.204441 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.204441 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.204511 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.204752 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.204752 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.204801 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.204936 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.205077 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"mutex_wait_us":35,"thread_start_us":83,"threads_started":1}
I20260926 05:42:00.205885 30510 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.23.213.252:40177: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "b3f821498f104da68674c7532371e0b4"
member_type: NON_VOTER
last_known_addr {
host: "127.23.213.252"
port: 40177
}
attrs {
promote: true
}
}
dest_uuid: "dccae96291a9413783a14543383a29ca"
cas_config_opid_index: 5
I20260926 05:42:00.206139 30510 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } }
I20260926 05:42:00.206490 31069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3f821498f104da68674c7532371e0b4 (127.23.213.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.206614 31121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 LEARNER]: Refusing update from remote peer dccae96291a9413783a14543383a29ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 05:42:00.206801 31070 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 05:42:00.207381 31121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER b3f821498f104da68674c7532371e0b4 (127.23.213.252) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.207595 31069 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.23.213.252:40177
I20260926 05:42:00.207667 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.207733 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:00.207934 31204 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:00.208509 31151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: NON_VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: true } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.208586 31151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:42:00.208892 31204 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Refusing update from remote peer dccae96291a9413783a14543383a29ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 05:42:00.209096 31147 master.cc:537] Master@127.23.213.252:40177 shutting down...
I20260926 05:42:00.209206 31071 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:00.209481 31071 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: attempting to promote NON_VOTER 4fa2426091a94d768868fdf7ed313b13 to VOTER
I20260926 05:42:00.209690 31071 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } }
I20260926 05:42:00.210376 31121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 LEARNER]: Refusing update from remote peer dccae96291a9413783a14543383a29ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20260926 05:42:00.210990 30496 consensus_peers.cc:487] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca -> Peer b3f821498f104da68674c7532371e0b4 (127.23.213.252:40177): Couldn't send request to peer b3f821498f104da68674c7532371e0b4. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 05:42:00.211325 31069 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 05:42:00.212148 31070 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.212235 31147 raft_consensus.cc:2229] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:00.212311 31147 raft_consensus.cc:2258] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:00.212361 31147 tablet_replica.cc:330] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: stopping tablet replica
I20260926 05:42:00.212683 31121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.213029 31070 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.213093 31070 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:00.213137 31070 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.213173 31070 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:00.213599 31151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.213685 31151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:42:00.214680 30463 connection.cc:535] client connection to 127.23.213.252:40177 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:42:00.226311 31147 master.cc:552] Master@127.23.213.252:40177 shutdown complete.
I20260926 05:42:00.226392 31147 master_runner.cc:305] Clearing existing system tablet
I20260926 05:42:00.226555 31147 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:00.226621 31147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:00.227296 31147 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:42:00.227850 31259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.228049 31147 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:00.228134 31147 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:42:00.228161 31147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
uuid: "b3f821498f104da68674c7532371e0b4"
format_stamp: "Formatted at 2026-09-26 05:42:00 on dist-test-slave-1fdt"
server_key: "cb8ea994a8e025806614bcb585bb78d1"
server_key_iv: "4dcaafadaf14b3be7455be9411b93835"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.228976 31147 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:00.230451 31147 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 05:42:00.230525 31147 log.cc:1196] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 05:42:00.230724 31147 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Deleting consensus metadata
I20260926 05:42:00.230933 31147 master_runner.cc:315] Copying system tablet from 127.23.213.254:35903
I20260926 05:42:00.231551 31147 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: tablet copy: Beginning tablet copy session from remote peer at address 127.23.213.254:35903
I20260926 05:42:00.237799 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.231951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57234 (local address 127.23.213.254:35903)
0926 05:42:00.232018 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.232020 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.232135 (+ 115us) server_negotiation.cc:407] Connection header received
0926 05:42:00.232219 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.232220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.232230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.232308 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.232647 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.233162 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.233985 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.234224 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.234312 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.235319 (+ 1007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.235320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.235321 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.235328 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.236655 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.236995 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.236996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.236997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.237057 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.237261 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.237262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.237262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.237377 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.237462 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.237612 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.237686 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:42:00.237824 31265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.231855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57234)
0926 05:42:00.232052 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.232065 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.232154 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.232314 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.232315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.232499 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.232566 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.232567 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.233246 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.233247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.233901 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.233903 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.234315 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.234316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.234393 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.234876 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.234880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.235212 (+ 332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.236746 (+ 1534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.236747 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.236747 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.236925 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.237123 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.237124 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.237124 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.237196 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.237452 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.237453 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.237497 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.237584 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.237685 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":69,"threads_started":1}
I20260926 05:42:00.238178 30530 tablet_copy_service.cc:140] P dccae96291a9413783a14543383a29ca: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer b3f821498f104da68674c7532371e0b4 ({username='oryx', principal='oryx/127.23.213.252@KRBTEST.COM'} at 127.0.0.1:57234)
I20260926 05:42:00.238233 30530 tablet_copy_service.cc:161] P dccae96291a9413783a14543383a29ca: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer b3f821498f104da68674c7532371e0b4 at {username='oryx', principal='oryx/127.23.213.252@KRBTEST.COM'} at 127.0.0.1:57234: session id = b3f821498f104da68674c7532371e0b4-00000000000000000000000000000000
I20260926 05:42:00.238673 30530 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 05:42:00.239156 31147 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 05:42:00.240150 31147 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: tablet copy: Starting download of 0 data blocks...
I20260926 05:42:00.240303 31147 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: tablet copy: Starting download of 1 WAL segments...
I20260926 05:42:00.241067 31147 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 05:42:00.241863 30530 tablet_copy_service.cc:342] P dccae96291a9413783a14543383a29ca: Request end of tablet copy session b3f821498f104da68674c7532371e0b4-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.23.213.252@KRBTEST.COM'} at 127.0.0.1:57234
I20260926 05:42:00.241927 30530 tablet_copy_service.cc:433] P dccae96291a9413783a14543383a29ca: ending tablet copy session b3f821498f104da68674c7532371e0b4-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer b3f821498f104da68674c7532371e0b4
I20260926 05:42:00.244279 31147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:00.245519 31266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.245543 31267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.245553 31269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:00.245877 31147 server_base.cc:1034] running on GCE node
I20260926 05:42:00.245987 31147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:00.246172 31147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:00.247306 31147 hybrid_clock.cc:648] HybridClock initialized: now 1790401320247296 us; error 25 us; skew 500 ppm
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:00.248767 31147 init.cc:377] Logged in from keytab as oryx/127.23.213.252@KRBTEST.COM (short username oryx)
I20260926 05:42:00.249037 31147 webserver.cc:460] Webserver started at http://127.23.213.252:46211/ using document root <none> and password file <none>
I20260926 05:42:00.249173 31147 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:00.249210 31147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:00.249979 31147 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:42:00.250634 31276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.250801 31147 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:42:00.250860 31147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
uuid: "b3f821498f104da68674c7532371e0b4"
format_stamp: "Formatted at 2026-09-26 05:42:00 on dist-test-slave-1fdt"
server_key: "cb8ea994a8e025806614bcb585bb78d1"
server_key_iv: "4dcaafadaf14b3be7455be9411b93835"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.250922 31147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:00.272090 31147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:00.273933 31147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:00.277030 31147 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.252:40177
I20260926 05:42:00.277307 31328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.252:40177 every 8 connection(s)
I20260926 05:42:00.277428 31147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
I20260926 05:42:00.278205 31329 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:42:00.278731 31329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Bootstrap starting.
I20260926 05:42:00.281621 31329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:42:00.281883 31329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Bootstrap complete.
I20260926 05:42:00.282087 31329 raft_consensus.cc:348] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } }
I20260926 05:42:00.282173 31329 raft_consensus.cc:729] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: b3f821498f104da68674c7532371e0b4, State: Initialized, Role: LEARNER
I20260926 05:42:00.282223 31329 consensus_queue.cc:260] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } }
I20260926 05:42:00.282393 31329 sys_catalog.cc:564] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:00.282393 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.282486 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: This master's current role is: LEARNER
I20260926 05:42:00.283818 31343 catalog_manager.cc:1259] Loaded cluster ID: b9443604a1104cce9a2871a783d8a847
I20260926 05:42:00.283864 31343 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: loading cluster ID for follower catalog manager: success
I20260926 05:42:00.284866 31343 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: acquiring CA information for follower catalog manager: success
I20260926 05:42:00.285149 31343 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:42:00.323961 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.315879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.23.213.252:40177)
0926 05:42:00.316015 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.316017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.316033 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:42:00.316058 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.316059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.316069 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.316209 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.317109 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.318059 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.319119 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.319449 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.319520 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.321289 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.321290 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.321290 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.321296 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.322844 (+ 1548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.323243 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.323244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.323244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.323295 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.323473 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.323474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.323474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.323581 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.323663 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.323783 (+ 120us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.323842 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
I20260926 05:42:00.324491 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30572
I20260926 05:42:00.330751 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:33691
--local_ip_for_outbound_sockets=127.23.213.193
--tserver_master_addrs=127.23.213.254:35903
--webserver_port=35531
--webserver_interface=127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab
--principal=oryx/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:42:00.343171 30835 heartbeater.cc:503] Master 127.23.213.254:35903 was elected leader, sending a full tablet report...
W20260926 05:42:00.413388 31345 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:00.413604 31345 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:00.413628 31345 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:00.414793 31345 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:00.414836 31345 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:00.414863 31345 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:42:00.416498 31345 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab
--principal=oryx/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:33691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=35531
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31345
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:00.416744 31345 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:00.416939 31345 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:00.422771 31353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.422766 31351 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.422792 31350 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:00.423135 31345 server_base.cc:1034] running on GCE node
I20260926 05:42:00.423352 31345 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:00.423556 31345 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:00.424746 31345 hybrid_clock.cc:648] HybridClock initialized: now 1790401320424728 us; error 30 us; skew 500 ppm
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:00.427289 31345 init.cc:377] Logged in from keytab as oryx/127.23.213.193@KRBTEST.COM (short username oryx)
I20260926 05:42:00.427743 31345 webserver.cc:460] Webserver started at http://127.23.213.193:35531/ using document root <none> and password file <none>
I20260926 05:42:00.427943 31345 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:00.427994 31345 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:00.429280 31345 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:42:00.429893 31360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.430048 31345 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:42:00.430137 31345 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "e2add9c82ce14ffeb3db41d11218936a"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "a4ef7babd7027d79a2abf01d62e5a9d5"
server_key_iv: "adf16d7c8bbb9e672c34b06ec390009d"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.430462 31345 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:00.465842 31345 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:00.468266 31345 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:00.468380 31345 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:00.468583 31345 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:00.468911 31345 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:00.468955 31345 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.469002 31345 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:00.469026 31345 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.474974 31345 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:33691
I20260926 05:42:00.475018 31473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:33691 every 8 connection(s)
I20260926 05:42:00.475400 31345 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:42:00.476143 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 31345
I20260926 05:42:00.476305 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30706
I20260926 05:42:00.482110 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:36013
--local_ip_for_outbound_sockets=127.23.213.194
--tserver_master_addrs=127.23.213.254:35903
--webserver_port=42037
--webserver_interface=127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab
--principal=oryx/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.253@KRBTEST.COM
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.252@KRBTEST.COM
I20260926 05:42:00.491091 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.476361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:52759 (local address 127.23.213.253:39357)
0926 05:42:00.477221 (+ 860us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.477223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.479174 (+ 1951us) server_negotiation.cc:407] Connection header received
0926 05:42:00.480306 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.480308 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.480321 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.480355 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.481521 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.482333 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484007 (+ 1674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.484308 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484349 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.487471 (+ 3122us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.487472 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.487472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.487479 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.489287 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.489862 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.489863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.489863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.489921 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.490277 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.490278 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.490278 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.490420 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.490472 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.490828 (+ 356us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.490914 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":832}
I20260926 05:42:00.491672 31478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.476152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.23.213.193:52759)
0926 05:42:00.479008 (+ 2856us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.479056 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.480174 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.480554 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.480559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.481076 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.481400 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.481406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.482714 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.482715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.483905 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.483908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.484440 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.484600 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.485296 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.485307 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.487351 (+ 2044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.489420 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.489426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.489428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.489769 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.490015 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.490016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.490016 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.490195 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.490503 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.490510 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.490672 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.491227 (+ 555us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.491415 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2806,"mutex_wait_us":13,"thread_start_us":2627,"threads_started":1}
I20260926 05:42:00.492172 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.476375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:34489 (local address 127.23.213.254:35903)
0926 05:42:00.476769 (+ 394us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.476771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.479203 (+ 2432us) server_negotiation.cc:407] Connection header received
0926 05:42:00.480308 (+ 1105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.480309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.480322 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.480355 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.481521 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.482333 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484006 (+ 1673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.484308 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484349 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.488637 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.488639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.488639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.488647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.490381 (+ 1734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.490795 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.490796 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.490796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.490854 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.491337 (+ 483us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.491338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.491339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.491478 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.491539 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.491759 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.492077 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365}
I20260926 05:42:00.492197 31476 heartbeater.cc:347] Connected to a master server at 127.23.213.253:39357
I20260926 05:42:00.492282 31476 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.492460 31476 heartbeater.cc:511] Master 127.23.213.253:39357 requested a full tablet report, sending...
I20260926 05:42:00.492635 31480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.476152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:34489)
0926 05:42:00.479008 (+ 2856us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.479056 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.480174 (+ 1118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.480554 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.480559 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.481075 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.481400 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.481406 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.482461 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.482462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.483905 (+ 1443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.483908 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.484444 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.484600 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.485502 (+ 902us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.485507 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.488261 (+ 2754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.490501 (+ 2240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.490501 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.490502 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.490707 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.490929 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.490930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.490930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.491255 (+ 325us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.491599 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.491600 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.491637 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.492259 (+ 622us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.492365 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2809,"mutex_wait_us":78,"thread_start_us":2624,"threads_started":1}
I20260926 05:42:00.492965 31111 ts_manager.cc:194] Registered new tserver with Master: e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691)
I20260926 05:42:00.493108 31475 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35903
I20260926 05:42:00.493170 31475 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.493237 31475 heartbeater.cc:511] Master 127.23.213.254:35903 requested a full tablet report, sending...
I20260926 05:42:00.493490 30510 ts_manager.cc:194] Re-registered known tserver with Master: e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691)
I20260926 05:42:00.494026 30510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:34489
I20260926 05:42:00.494582 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.476797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:59919 (local address 127.23.213.252:40177)
0926 05:42:00.476841 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.476842 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.479334 (+ 2492us) server_negotiation.cc:407] Connection header received
0926 05:42:00.480319 (+ 985us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.480321 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.480338 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.480368 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.481523 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.482316 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484011 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.484340 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484385 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.490708 (+ 6323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.490710 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.490710 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.490719 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.492783 (+ 2064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.493526 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.493527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.493527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.493601 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.493878 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.493879 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.493879 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.494021 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.494067 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.494345 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.494441 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:42:00.494882 31479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.476151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.23.213.193:59919)
0926 05:42:00.479279 (+ 3128us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.479288 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.480195 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.480556 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.480559 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.481107 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.481400 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.481407 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.482456 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.482457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.483907 (+ 1450us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.483910 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.484467 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.484468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.484600 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.485360 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.485366 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.490597 (+ 5231us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.493102 (+ 2505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.493103 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.493104 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.493431 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.493719 (+ 288us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.493720 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.493721 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.493802 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.494171 (+ 369us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.494172 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.494217 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.494661 (+ 444us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.494777 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3100,"mutex_wait_us":235,"thread_start_us":2504,"threads_started":1}
I20260926 05:42:00.495257 31474 heartbeater.cc:347] Connected to a master server at 127.23.213.252:40177
I20260926 05:42:00.495326 31474 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.495397 31474 heartbeater.cc:511] Master 127.23.213.252:40177 requested a full tablet report, sending...
I20260926 05:42:00.495757 31293 ts_manager.cc:194] Registered new tserver with Master: e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691)
I20260926 05:42:00.500859 30969 heartbeater.cc:503] Master 127.23.213.254:35903 was elected leader, sending a full tablet report...
W20260926 05:42:00.569298 31481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:00.569527 31481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:00.569566 31481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:00.570827 31481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:00.570890 31481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:00.570935 31481 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:42:00.572419 31481 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab
--principal=oryx/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:36013
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=42037
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:00.572679 31481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:00.572906 31481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:00.578080 31486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.578096 31487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.578490 31489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:00.578539 31481 server_base.cc:1034] running on GCE node
I20260926 05:42:00.578805 31481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:00.579154 31481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:00.580330 31481 hybrid_clock.cc:648] HybridClock initialized: now 1790401320580302 us; error 43 us; skew 500 ppm
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:00.583544 31481 init.cc:377] Logged in from keytab as oryx/127.23.213.194@KRBTEST.COM (short username oryx)
I20260926 05:42:00.584028 31481 webserver.cc:460] Webserver started at http://127.23.213.194:42037/ using document root <none> and password file <none>
I20260926 05:42:00.584271 31481 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:00.584354 31481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:00.585845 31481 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:42:00.586454 31496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.586618 31481 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:00.586721 31481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "327e803d0d744eb8933eff3fc78e4d2d"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "ec38b67031f397320ce637d09e2f4afe"
server_key_iv: "c678796554833661d457d89a1146a673"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.587117 31481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:00.603048 31481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:00.604944 31481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:00.605118 31481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:00.605358 31481 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:00.605661 31481 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:00.605713 31481 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.605779 31481 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:00.605824 31481 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.612207 31481 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:36013
I20260926 05:42:00.612258 31609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:36013 every 8 connection(s)
I20260926 05:42:00.612589 31481 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:42:00.617542 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 31481
I20260926 05:42:00.617707 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30840
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.252@KRBTEST.COM
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.253@KRBTEST.COM
I20260926 05:42:00.625720 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:38665
--local_ip_for_outbound_sockets=127.23.213.195
--tserver_master_addrs=127.23.213.254:35903
--webserver_port=42363
--webserver_interface=127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab
--principal=oryx/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:33533
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:42:00.625916 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.613315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:38285 (local address 127.23.213.252:40177)
0926 05:42:00.613362 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.613364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.613888 (+ 524us) server_negotiation.cc:407] Connection header received
0926 05:42:00.614742 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.614743 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.614754 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.614787 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.615647 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.616374 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.617532 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.617880 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.617925 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.622272 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.622274 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.622274 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.622282 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.624070 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.624764 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.624765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.624766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.624821 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.625121 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.625122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.625122 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.625270 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.625314 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.625695 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.625809 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:42:00.625926 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.613769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:44159 (local address 127.23.213.253:39357)
0926 05:42:00.613816 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.613819 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.614000 (+ 181us) server_negotiation.cc:407] Connection header received
0926 05:42:00.614740 (+ 740us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.614742 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.614754 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.614787 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.615478 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.616060 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.617535 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.618760 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.618802 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.622382 (+ 3580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.622383 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.622384 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.622391 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.624212 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.624842 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.624844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.624844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.624903 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.625122 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.625123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.625123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.625274 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.625317 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.625696 (+ 379us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.625803 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:42:00.625945 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.613316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:41419 (local address 127.23.213.254:35903)
0926 05:42:00.613579 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.613582 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.613593 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:42:00.614717 (+ 1124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.614719 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.614731 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.614788 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.615463 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.616265 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.617562 (+ 1297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.617951 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.617991 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.621770 (+ 3779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.621772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.621772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.621780 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.624477 (+ 2697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.625122 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.625123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.625124 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.625173 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.625376 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.625377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.625377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.625509 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.625550 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.625784 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.625857 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":233}
I20260926 05:42:00.626598 31614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.613666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.23.213.194:44159)
0926 05:42:00.613938 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.613945 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.614633 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.614873 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.614878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.615246 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.615364 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.615366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.616159 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.616160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.617440 (+ 1280us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.617441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.618923 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.618924 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.618991 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.619423 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.619428 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.622270 (+ 2842us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.624301 (+ 2031us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.624302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.624302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.624760 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.624972 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.624973 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.624974 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.625054 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.625362 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.625378 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.625536 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.626053 (+ 517us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.626278 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":31,"thread_start_us":87,"threads_started":1}
I20260926 05:42:00.626597 31616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.613324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.23.213.194:38285)
0926 05:42:00.613805 (+ 481us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.613815 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.614633 (+ 818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.614873 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.614878 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.615238 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.615362 (+ 124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.615367 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.616449 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.616450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.617422 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.617425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.618012 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.618013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.618156 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.618906 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.618917 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.622161 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.624253 (+ 2092us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.624261 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.624264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.624696 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.624902 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.624903 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.624903 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.625042 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.625362 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.625368 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.625534 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.626052 (+ 518us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.626377 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":454,"mutex_wait_us":155,"thread_start_us":103,"threads_started":1}
I20260926 05:42:00.627282 31610 heartbeater.cc:347] Connected to a master server at 127.23.213.252:40177
I20260926 05:42:00.627404 31610 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.627496 31612 heartbeater.cc:347] Connected to a master server at 127.23.213.253:39357
I20260926 05:42:00.627555 31612 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.627615 31610 heartbeater.cc:511] Master 127.23.213.252:40177 requested a full tablet report, sending...
I20260926 05:42:00.627627 31612 heartbeater.cc:511] Master 127.23.213.253:39357 requested a full tablet report, sending...
I20260926 05:42:00.627281 31615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.613324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:41419)
0926 05:42:00.613494 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.613528 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.614607 (+ 1079us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.614874 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.614878 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.615167 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.615359 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.615363 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.616345 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.616345 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.617433 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.617436 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.618062 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.618063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.618156 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.618923 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.618927 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.621648 (+ 2721us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.624570 (+ 2922us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.624571 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.624571 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.625032 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.625246 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.625246 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.625247 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.625303 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.625598 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.625598 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.625635 (+ 37us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.627046 (+ 1411us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.627153 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":3,"thread_start_us":105,"threads_started":1}
I20260926 05:42:00.628095 31293 ts_manager.cc:194] Registered new tserver with Master: 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194:36013)
I20260926 05:42:00.628142 31111 ts_manager.cc:194] Registered new tserver with Master: 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194:36013)
I20260926 05:42:00.628338 31611 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35903
I20260926 05:42:00.628382 31611 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.628453 31611 heartbeater.cc:511] Master 127.23.213.254:35903 requested a full tablet report, sending...
I20260926 05:42:00.628693 30510 ts_manager.cc:194] Re-registered known tserver with Master: 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194:36013)
I20260926 05:42:00.629068 30510 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:41419
I20260926 05:42:00.651799 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.648776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.0.0.1:40280)
0926 05:42:00.648843 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.648857 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.648925 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.649110 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.649111 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.649112 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:00.649161 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.649164 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.650091 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.650092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.650848 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.650850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.651610 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.651610 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.651657 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.651660 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.651705 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.651739 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 05:42:00.651818 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.648883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.23.213.252:40177)
0926 05:42:00.648962 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.648964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.648974 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:00.649004 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.649005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.649029 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.649100 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:00.649236 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.650014 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.650935 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.651531 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.651606 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.651669 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.651712 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.651749 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:42:00.652176 31303 raft_consensus.cc:1203] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 05:42:00.652340 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dccae96291a9413783a14543383a29ca. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.652474 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: This master's current role is: LEARNER
W20260926 05:42:00.711076 31617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:00.711310 31617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:00.711349 31617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:00.712520 31617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:00.712595 31617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:00.712641 31617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:42:00.714286 31617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:33533
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab
--principal=oryx/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:38665
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=42363
--tserver_master_addrs=127.23.213.254:35903,127.23.213.253:39357,127.23.213.252:40177
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:00.714560 31617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:00.714790 31617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:00.720054 31623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.720046 31622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:00.720046 31625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:00.720280 31617 server_base.cc:1034] running on GCE node
I20260926 05:42:00.720482 31617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:00.720678 31617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:00.721797 31617 hybrid_clock.cc:648] HybridClock initialized: now 1790401320721780 us; error 31 us; skew 500 ppm
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:00.724159 31617 init.cc:377] Logged in from keytab as oryx/127.23.213.195@KRBTEST.COM (short username oryx)
I20260926 05:42:00.724644 31617 webserver.cc:460] Webserver started at http://127.23.213.195:42363/ using document root <none> and password file <none>
I20260926 05:42:00.724900 31617 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:00.724965 31617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:00.726516 31617 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:42:00.727205 31632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.727428 31617 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:00.727492 31617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "31bf19fb701148f58f69c8b548f2cbc5"
format_stamp: "Formatted at 2026-09-26 05:41:59 on dist-test-slave-1fdt"
server_key: "2d60e63e3b4d1063236babb91e4f9f83"
server_key_iv: "647ddd049c6b7a6fdc8838dc51091af3"
server_key_version: "encryptionkey@0"
I20260926 05:42:00.727843 31617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:00.761709 31617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:00.763893 31617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:00.764057 31617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:00.764343 31617 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:00.764671 31617 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:00.764729 31617 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.764770 31617 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:00.764797 31617 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:00.770714 31617 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:38665
I20260926 05:42:00.770761 31745 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:38665 every 8 connection(s)
I20260926 05:42:00.771121 31617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:42:00.772265 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 31617
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.252@KRBTEST.COM
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.253@KRBTEST.COM
Sep 26 05:42:00 dist-test-slave-1fdt krb5kdc[30465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401320, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:00.784382 31757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.773330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57244 (local address 127.23.213.254:35903)
0926 05:42:00.773632 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.773634 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.773909 (+ 275us) server_negotiation.cc:407] Connection header received
0926 05:42:00.774014 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.774015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.774029 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.774102 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.774823 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.775635 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.776673 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.776978 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.777016 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.779621 (+ 2605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.779622 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.779622 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.779630 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.782271 (+ 2641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.783448 (+ 1177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.783449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.783450 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.783504 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.783741 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.783742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.783743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.783864 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.783939 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.784151 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.784256 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":29,"server-negotiator.queue_time_us":276,"thread_start_us":55,"threads_started":1}
I20260926 05:42:00.784647 31758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.773367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40294 (local address 127.23.213.252:40177)
0926 05:42:00.774046 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.774048 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.774180 (+ 132us) server_negotiation.cc:407] Connection header received
0926 05:42:00.774213 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.774214 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.774223 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.774301 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.774988 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.775717 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.776875 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.777177 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.777234 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.778995 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.778996 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.778996 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.779021 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.781830 (+ 2809us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.782239 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.782240 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.782241 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.782337 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.783396 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.783397 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.783398 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.783545 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.783654 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.784453 (+ 799us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.784537 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":49,"threads_started":1}
I20260926 05:42:00.786813 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.771884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:39099 (local address 127.23.213.253:39357)
0926 05:42:00.771923 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.771924 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.772178 (+ 254us) server_negotiation.cc:407] Connection header received
0926 05:42:00.772981 (+ 803us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.772982 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.772994 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.773043 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.773949 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.774740 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.777936 (+ 3196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.778274 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.778319 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.781508 (+ 3189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.781515 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.781515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.781523 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.783474 (+ 1951us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.785701 (+ 2227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.785702 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.785703 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.785778 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.786047 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.786048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.786048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.786171 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.786215 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.786601 (+ 386us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.786681 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:42:00.787009 30550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.771837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:39165 (local address 127.23.213.254:35903)
0926 05:42:00.771890 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.771892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.772178 (+ 286us) server_negotiation.cc:407] Connection header received
0926 05:42:00.772821 (+ 643us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.772823 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.772838 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.772916 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.774823 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.775547 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.778695 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.778992 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.779064 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.781769 (+ 2705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.781770 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.781771 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.781778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.784074 (+ 2296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.785622 (+ 1548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.785623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.785623 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.785685 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.786191 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.786192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.786192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.786327 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.786368 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.786836 (+ 468us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.786918 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:42:00.787319 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.771833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.23.213.195:39099)
0926 05:42:00.772039 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.772088 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.772696 (+ 608us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.773091 (+ 395us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.773092 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.773521 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.773831 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.773839 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.775670 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.775671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.776935 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.776937 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.778395 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.778396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.778482 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.779149 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.779153 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.781403 (+ 2250us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.784351 (+ 2948us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.784359 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.784362 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.785625 (+ 1263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.785856 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.785856 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.785857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.785964 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.786265 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.786271 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.786458 (+ 187us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.786578 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.786740 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":89,"threads_started":1}
I20260926 05:42:00.787663 31751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.771833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:39165)
0926 05:42:00.772039 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.772088 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.772698 (+ 610us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.773013 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.773017 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.773668 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.773831 (+ 163us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.773838 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.777002 (+ 3164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.777002 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.777834 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.777837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.779776 (+ 1939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.779777 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.779850 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.780256 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.780261 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.781644 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.784395 (+ 2751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.784396 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.784396 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.785541 (+ 1145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.785787 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.785788 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.785788 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.786099 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.786418 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.786419 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.786463 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.787153 (+ 690us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.787524 (+ 371us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":20,"thread_start_us":64,"threads_started":1}
I20260926 05:42:00.787961 31748 heartbeater.cc:347] Connected to a master server at 127.23.213.253:39357
I20260926 05:42:00.788060 31748 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.788247 31748 heartbeater.cc:511] Master 127.23.213.253:39357 requested a full tablet report, sending...
I20260926 05:42:00.786458 30510 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57244:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:42:00.788684 31111 ts_manager.cc:194] Registered new tserver with Master: 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195:38665)
W20260926 05:42:00.788828 30510 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:00.789295 31747 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35903
I20260926 05:42:00.789350 31747 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.789410 31747 heartbeater.cc:511] Master 127.23.213.254:35903 requested a full tablet report, sending...
I20260926 05:42:00.789623 30509 ts_manager.cc:194] Re-registered known tserver with Master: 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195:38665)
I20260926 05:42:00.789742 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: NON_VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: true } } }
I20260926 05:42:00.789858 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:00.790143 30509 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:39165
I20260926 05:42:00.790248 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.773765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.23.213.253:39357)
0926 05:42:00.773864 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.773866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.778224 (+ 4358us) server_negotiation.cc:407] Connection header received
0926 05:42:00.778271 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.778272 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.778286 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.778356 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.780209 (+ 1853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.780985 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.783683 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.783996 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.784053 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.786288 (+ 2235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.786289 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.786289 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.786296 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.788354 (+ 2058us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.788993 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.788995 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.788995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.789046 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.789260 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.789260 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.789261 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.789404 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.789506 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.789819 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.790126 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":53,"threads_started":1}
I20260926 05:42:00.791853 31764 raft_consensus.cc:1050] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: attempting to promote NON_VOTER b3f821498f104da68674c7532371e0b4 to VOTER
I20260926 05:42:00.791961 31764 consensus_queue.cc:237] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.9, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:00.792425 31120 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Refusing update from remote peer dccae96291a9413783a14543383a29ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260926 05:42:00.792690 31765 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 05:42:00.792924 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.772170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.23.213.195:38769)
0926 05:42:00.775025 (+ 2855us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.775035 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.775102 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.775258 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.775259 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.775410 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:00.775463 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.775464 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.776373 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.776374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.777203 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.777205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.777727 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.777728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.777906 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.778470 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:00.778480 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:00.781059 (+ 2579us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:00.784757 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.784758 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.784759 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.791725 (+ 6966us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.792015 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:00.792016 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:00.792017 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:00.792109 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:00.792402 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:00.792403 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:00.792449 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.792510 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.792624 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2829,"thread_start_us":60,"threads_started":1}
I20260926 05:42:00.792922 31121 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Deduplicated request from leader. Original: 1.8->[1.9-1.10] Dedup: 1.9->[1.10-1.10]
I20260926 05:42:00.793145 31303 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 LEARNER]: Refusing update from remote peer dccae96291a9413783a14543383a29ca: Log matching property violated. Preceding OpId in replica: term: 1 index: 9. Preceding OpId from leader: term: 1 index: 10. (index mismatch)
I20260926 05:42:00.793283 31069 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [term 1 LEADER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b3f821498f104da68674c7532371e0b4 (127.23.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.793334 31764 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 05:42:00.793423 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.772339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:38769 (local address 127.23.213.252:40177)
0926 05:42:00.773192 (+ 853us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.773194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.775079 (+ 1885us) server_negotiation.cc:407] Connection header received
0926 05:42:00.775167 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.775168 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.775177 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.775205 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.775539 (+ 334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.776281 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.777297 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.777584 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.777639 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.781820 (+ 4181us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:00.781821 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:00.781822 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:00.781830 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:00.784258 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.791809 (+ 7551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.791810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.791811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.791878 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:00.792177 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:00.792178 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:00.792178 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:00.792340 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:00.792406 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.793263 (+ 857us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.793350 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824}
I20260926 05:42:00.793898 31746 heartbeater.cc:347] Connected to a master server at 127.23.213.252:40177
I20260926 05:42:00.793954 31746 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:00.793880 31303 raft_consensus.cc:2932] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b3f821498f104da68674c7532371e0b4 (127.23.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.794052 31746 heartbeater.cc:511] Master 127.23.213.252:40177 requested a full tablet report, sending...
I20260926 05:42:00.794351 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.794407 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:00.794672 31069 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.794749 31069 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:00.794763 31121 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Committing config change with OpId 1.10: config changed from index 7 to 10, b3f821498f104da68674c7532371e0b4 (127.23.213.252) changed from NON_VOTER to VOTER. New config: { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.795019 31293 ts_manager.cc:194] Registered new tserver with Master: 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195:38665)
I20260926 05:42:00.795409 31332 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.795504 31332 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:42:00.796875 31776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.796962 31776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:42:00.799880 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.795416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:33691 (local address 127.0.0.1:52662)
0926 05:42:00.795712 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.795725 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.795771 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.796154 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.796155 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.796157 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:00.796205 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.796206 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.797581 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.797583 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.798839 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.798841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.799659 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.799660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.799719 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.799723 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.799776 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.799815 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279}
I20260926 05:42:00.799933 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.795717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52662 (local address 127.23.213.193:33691)
0926 05:42:00.795956 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.795961 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.795969 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:00.796004 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.796005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.796041 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.796133 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:00.796284 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.797485 (+ 1201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.798946 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.799561 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.799678 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.799739 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.799793 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.799832 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":200,"thread_start_us":70,"threads_started":1}
I20260926 05:42:00.800132 31774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.795458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38665 (local address 127.0.0.1:41226)
0926 05:42:00.796219 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.796228 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.796290 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.796711 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.796712 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.796714 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:00.796755 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.796756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.797943 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.797944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.798941 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.798944 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.799895 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.799896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.799957 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.799961 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.800009 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.800048 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"mutex_wait_us":37,"thread_start_us":71,"threads_started":1}
I20260926 05:42:00.800307 31773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.795502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:36013 (local address 127.0.0.1:57976)
0926 05:42:00.796363 (+ 861us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.796370 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.796417 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.796590 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.796590 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.796592 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:00.796633 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.796635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.797991 (+ 1356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.797992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.799152 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.799154 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.800092 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.800093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.800149 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.800153 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.800215 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.800251 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":847,"mutex_wait_us":32,"thread_start_us":82,"threads_started":1}
I20260926 05:42:00.800349 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.795628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41226 (local address 127.23.213.195:38665)
0926 05:42:00.796516 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.796521 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.796530 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:00.796563 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.796565 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.796581 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.796655 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:00.796813 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.797848 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.799027 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.799724 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.799812 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.799859 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.800241 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.800276 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":857,"thread_start_us":73,"threads_started":1}
I20260926 05:42:00.800530 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.795917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57976 (local address 127.23.213.194:36013)
0926 05:42:00.796112 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.796116 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.796446 (+ 330us) server_negotiation.cc:407] Connection header received
0926 05:42:00.796482 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.796484 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.796497 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.796574 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:00.796710 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.797917 (+ 1207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.799233 (+ 1316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.799931 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.800015 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.800060 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.800418 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.800461 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":94,"threads_started":1}
I20260926 05:42:00.800976 31408 tablet_service.cc:1467] Processing CreateTablet for tablet 8d388c66a7f749bfa05d59ad174cc597 (DEFAULT_TABLE table=test-table [id=6d05f49adcbf4d2fbdea6da9be4231b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:00.801153 31680 tablet_service.cc:1467] Processing CreateTablet for tablet 8d388c66a7f749bfa05d59ad174cc597 (DEFAULT_TABLE table=test-table [id=6d05f49adcbf4d2fbdea6da9be4231b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:00.801352 31408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d388c66a7f749bfa05d59ad174cc597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:00.801381 31680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d388c66a7f749bfa05d59ad174cc597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:00.801462 31544 tablet_service.cc:1467] Processing CreateTablet for tablet 8d388c66a7f749bfa05d59ad174cc597 (DEFAULT_TABLE table=test-table [id=6d05f49adcbf4d2fbdea6da9be4231b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:00.802112 31544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d388c66a7f749bfa05d59ad174cc597. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:00.803985 31779 tablet_bootstrap.cc:492] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Bootstrap starting.
I20260926 05:42:00.804522 31779 tablet_bootstrap.cc:654] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:00.804749 31779 log.cc:824] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:00.804946 31780 tablet_bootstrap.cc:492] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Bootstrap starting.
I20260926 05:42:00.805490 31782 tablet_bootstrap.cc:492] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Bootstrap starting.
I20260926 05:42:00.805572 31780 tablet_bootstrap.cc:654] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:00.805622 31779 tablet_bootstrap.cc:492] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: No bootstrap required, opened a new log
I20260926 05:42:00.805691 31779 ts_tablet_manager.cc:1397] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:00.805920 31780 log.cc:824] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:00.806149 31782 tablet_bootstrap.cc:654] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:00.806470 31782 log.cc:824] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:00.806715 31780 tablet_bootstrap.cc:492] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: No bootstrap required, opened a new log
I20260926 05:42:00.806782 31780 ts_tablet_manager.cc:1397] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:42:00.807309 31782 tablet_bootstrap.cc:492] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: No bootstrap required, opened a new log
I20260926 05:42:00.807375 31782 ts_tablet_manager.cc:1397] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:00.808081 31779 raft_consensus.cc:348] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.808231 31779 raft_consensus.cc:374] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:00.808274 31779 raft_consensus.cc:729] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bf19fb701148f58f69c8b548f2cbc5, State: Initialized, Role: FOLLOWER
I20260926 05:42:00.808383 31779 consensus_queue.cc:260] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.808666 31779 ts_tablet_manager.cc:1428] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:42:00.808799 31747 heartbeater.cc:503] Master 127.23.213.254:35903 was elected leader, sending a full tablet report...
I20260926 05:42:00.809096 31780 raft_consensus.cc:348] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.809259 31780 raft_consensus.cc:374] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:00.809301 31780 raft_consensus.cc:729] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2add9c82ce14ffeb3db41d11218936a, State: Initialized, Role: FOLLOWER
I20260926 05:42:00.809419 31780 consensus_queue.cc:260] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.809672 31780 ts_tablet_manager.cc:1428] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:42:00.809856 31475 heartbeater.cc:503] Master 127.23.213.254:35903 was elected leader, sending a full tablet report...
I20260926 05:42:00.810197 31782 raft_consensus.cc:348] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.810335 31782 raft_consensus.cc:374] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:00.810379 31782 raft_consensus.cc:729] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327e803d0d744eb8933eff3fc78e4d2d, State: Initialized, Role: FOLLOWER
I20260926 05:42:00.810488 31782 consensus_queue.cc:260] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.810737 31782 ts_tablet_manager.cc:1428] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:42:00.810824 31611 heartbeater.cc:503] Master 127.23.213.254:35903 was elected leader, sending a full tablet report...
W20260926 05:42:00.864007 31613 tablet.cc:2369] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:42:00.938508 31785 raft_consensus.cc:482] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:00.938627 31785 raft_consensus.cc:504] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.939033 31785 leader_election.cc:288] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691), 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194:36013)
I20260926 05:42:00.941968 31750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.939367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:36013 (local address 127.23.213.195:38395)
0926 05:42:00.939412 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.939422 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.939459 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.939653 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.939654 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.939656 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:00.939715 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.939718 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.940351 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.940352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.941132 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.941134 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.941697 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.941698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.941799 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.941802 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.941853 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.941880 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28,"mutex_wait_us":1}
I20260926 05:42:00.942008 31778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.939439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:38395 (local address 127.23.213.194:36013)
0926 05:42:00.939495 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.939497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.939505 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:00.939529 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.939530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.939539 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.939599 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:00.939790 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.940247 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.941218 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.941615 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.941670 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.941677 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.941872 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.941930 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:42:00.942082 31777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.939360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:58875 (local address 127.23.213.193:33691)
0926 05:42:00.939411 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.939413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.939423 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:00.939507 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.939508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.939520 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.939598 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:00.939747 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.940236 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.941264 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.941641 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.941731 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.941759 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.941953 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.942000 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:42:00.942095 31752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.939247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:33691 (local address 127.23.213.195:58875)
0926 05:42:00.939330 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:00.939346 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:00.939413 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:00.939606 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:00.939606 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:00.939608 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:00.939690 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.939692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.940311 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.940312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.941137 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:00.941140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.941780 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.941782 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:00.941877 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.941880 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:00.941935 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:42:00.941974 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":49}
I20260926 05:42:00.942294 31564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d388c66a7f749bfa05d59ad174cc597" candidate_uuid: "31bf19fb701148f58f69c8b548f2cbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327e803d0d744eb8933eff3fc78e4d2d" is_pre_election: true
I20260926 05:42:00.942413 31564 raft_consensus.cc:2454] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bf19fb701148f58f69c8b548f2cbc5 in term 0.
I20260926 05:42:00.942632 31634 leader_election.cc:302] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bf19fb701148f58f69c8b548f2cbc5, 327e803d0d744eb8933eff3fc78e4d2d; no voters:
I20260926 05:42:00.942694 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d388c66a7f749bfa05d59ad174cc597" candidate_uuid: "31bf19fb701148f58f69c8b548f2cbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2add9c82ce14ffeb3db41d11218936a" is_pre_election: true
I20260926 05:42:00.942833 31785 raft_consensus.cc:2781] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:00.942857 31428 raft_consensus.cc:2454] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31bf19fb701148f58f69c8b548f2cbc5 in term 0.
I20260926 05:42:00.942898 31785 raft_consensus.cc:482] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:00.942929 31785 raft_consensus.cc:3037] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:00.943662 31785 raft_consensus.cc:504] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.943801 31785 leader_election.cc:288] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [CANDIDATE]: Term 1 election: Requested vote from peers e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691), 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194:36013)
I20260926 05:42:00.943934 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d388c66a7f749bfa05d59ad174cc597" candidate_uuid: "31bf19fb701148f58f69c8b548f2cbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2add9c82ce14ffeb3db41d11218936a"
I20260926 05:42:00.943989 31564 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d388c66a7f749bfa05d59ad174cc597" candidate_uuid: "31bf19fb701148f58f69c8b548f2cbc5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "327e803d0d744eb8933eff3fc78e4d2d"
I20260926 05:42:00.944036 31428 raft_consensus.cc:3037] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:00.944085 31564 raft_consensus.cc:3037] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:00.944743 31428 raft_consensus.cc:2454] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bf19fb701148f58f69c8b548f2cbc5 in term 1.
I20260926 05:42:00.944746 31564 raft_consensus.cc:2454] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31bf19fb701148f58f69c8b548f2cbc5 in term 1.
I20260926 05:42:00.944900 31634 leader_election.cc:302] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31bf19fb701148f58f69c8b548f2cbc5, e2add9c82ce14ffeb3db41d11218936a; no voters:
I20260926 05:42:00.944983 31785 raft_consensus.cc:2781] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:00.945132 31785 raft_consensus.cc:686] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 LEADER]: Becoming Leader. State: Replica: 31bf19fb701148f58f69c8b548f2cbc5, State: Running, Role: LEADER
I20260926 05:42:00.945338 31785 consensus_queue.cc:237] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } }
I20260926 05:42:00.946198 30510 catalog_manager.cc:5495] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195). New cstate: current_term: 1 leader_uuid: "31bf19fb701148f58f69c8b548f2cbc5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } health_report { overall_health: UNKNOWN } } }
I20260926 05:42:00.946725 31764 sys_catalog.cc:455] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "dccae96291a9413783a14543383a29ca" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:00.946820 31764 sys_catalog.cc:458] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca [sys.catalog]: This master's current role is: LEADER
W20260926 05:42:00.978224 31477 tablet.cc:2369] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:42:00.979092 31775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.976566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41238 (local address 127.23.213.195:38665)
0926 05:42:00.976627 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.976629 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.976637 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:00.976701 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.976702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.976721 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.976791 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:00.976943 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.977430 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.978247 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.978501 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:00.978574 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:00.978745 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:00.978862 (+ 117us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:00.978953 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:00.978978 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 05:42:00.979000 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:42:00.980885 31564 raft_consensus.cc:1261] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 1 FOLLOWER]: Refusing update from remote peer 31bf19fb701148f58f69c8b548f2cbc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:00.980950 31428 raft_consensus.cc:1261] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 1 FOLLOWER]: Refusing update from remote peer 31bf19fb701148f58f69c8b548f2cbc5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:00.981284 31785 consensus_queue.cc:1035] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:00.981426 31788 consensus_queue.cc:1035] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:00.986426 31792 mvcc.cc:205] Tried to move back new op lower bound from 7333483810735087616 to 7333483810592956416. Current Snapshot: MvccSnapshot[applied={T|T < 7333483810735087616}]
I20260926 05:42:00.986558 31793 mvcc.cc:205] Tried to move back new op lower bound from 7333483810735087616 to 7333483810592956416. Current Snapshot: MvccSnapshot[applied={T|T < 7333483810735087616}]
I20260926 05:42:00.987159 30510 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:00.987246 30510 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57244:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:00.989002 30510 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dccae96291a9413783a14543383a29ca: Sending DeleteTablet for 3 replicas of tablet 8d388c66a7f749bfa05d59ad174cc597
I20260926 05:42:00.989301 31680 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d388c66a7f749bfa05d59ad174cc597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:00 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:41226
I20260926 05:42:00.989390 31408 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d388c66a7f749bfa05d59ad174cc597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:00 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:52662
I20260926 05:42:00.989604 31801 tablet_replica.cc:330] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: stopping tablet replica
I20260926 05:42:00.989643 31544 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d388c66a7f749bfa05d59ad174cc597 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:00 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:57976
I20260926 05:42:00.989811 31801 raft_consensus.cc:2229] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:42:00.989885 31802 tablet_replica.cc:330] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: stopping tablet replica
I20260926 05:42:00.989982 31801 raft_consensus.cc:2258] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:00.989998 31802 raft_consensus.cc:2229] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:00.990106 31802 raft_consensus.cc:2258] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:00.990278 31804 tablet_replica.cc:330] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: stopping tablet replica
I20260926 05:42:00.990378 31804 raft_consensus.cc:2229] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:00.990495 31804 raft_consensus.cc:2258] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:00.990646 31801 ts_tablet_manager.cc:1905] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:00.990753 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30478
I20260926 05:42:00.992070 31802 ts_tablet_manager.cc:1905] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:00.992468 31801 ts_tablet_manager.cc:1918] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:00.992606 31801 log.cc:1196] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/8d388c66a7f749bfa05d59ad174cc597
I20260926 05:42:00.992926 31801 ts_tablet_manager.cc:1939] T 8d388c66a7f749bfa05d59ad174cc597 P 31bf19fb701148f58f69c8b548f2cbc5: Deleting consensus metadata
I20260926 05:42:00.993052 31804 ts_tablet_manager.cc:1905] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:00.994810 31802 ts_tablet_manager.cc:1918] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:00.994939 31802 log.cc:1196] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/8d388c66a7f749bfa05d59ad174cc597
I20260926 05:42:00.994975 31804 ts_tablet_manager.cc:1918] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:00.995048 31804 log.cc:1196] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/8d388c66a7f749bfa05d59ad174cc597
I20260926 05:42:00.995311 31804 ts_tablet_manager.cc:1939] T 8d388c66a7f749bfa05d59ad174cc597 P 327e803d0d744eb8933eff3fc78e4d2d: Deleting consensus metadata
I20260926 05:42:00.995503 31802 ts_tablet_manager.cc:1939] T 8d388c66a7f749bfa05d59ad174cc597 P e2add9c82ce14ffeb3db41d11218936a: Deleting consensus metadata
W20260926 05:42:00.997463 31498 connection.cc:535] server connection from 127.0.0.1:57976 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 05:42:00.997519 31363 connection.cc:535] server connection from 127.0.0.1:52662 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 05:42:00.997516 31633 connection.cc:535] server connection from 127.0.0.1:41226 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:42:01.005729 31762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.990569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42092 (local address 127.23.213.253:39357)
0926 05:42:00.990612 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.990614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.997995 (+ 7381us) server_negotiation.cc:407] Connection header received
0926 05:42:00.998114 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.998115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.998131 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:00.998192 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:00.998867 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:00.999619 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:01.000506 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:01.000811 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:01.000877 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:01.002400 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:01.002401 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:01.002401 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:01.002409 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:01.004373 (+ 1964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:01.004752 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:01.004753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:01.004753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:01.004806 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:01.004987 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:01.004988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:01.004989 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:01.005104 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:01.005194 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:01.005494 (+ 300us) server_negotiation.cc:299] Negotiation successful
0926 05:42:01.005632 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:42:01.008203 31233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.999744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42096 (local address 127.23.213.253:39357)
0926 05:42:00.999786 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:00.999788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:00.999879 (+ 91us) server_negotiation.cc:407] Connection header received
0926 05:42:00.999938 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:00.999939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:00.999949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:01.000008 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:01.000726 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:01.001466 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:01.002530 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:01.002809 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:01.002849 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:01.004192 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:01.004193 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:01.004194 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:01.004201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:01.006693 (+ 2492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:01.007362 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:01.007364 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:01.007364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:01.007418 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:01.007626 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:01.007627 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:01.007627 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:01.007752 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:01.007817 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:01.008023 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 05:42:01.008086 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:42:01.008888 31344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:00.999664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40302 (local address 127.23.213.252:40177)
0926 05:42:01.000542 (+ 878us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:01.000544 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:01.000551 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:42:01.000572 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:01.000572 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:01.000583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:01.000627 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:01.002092 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:01.002811 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:01.003929 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:01.004239 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:01.004278 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:01.005935 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:01.005936 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:01.005936 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:01.005944 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:01.007683 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:01.008071 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:01.008072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:01.008072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:01.008131 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:01.008343 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:01.008344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:01.008344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:01.008461 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:01.008530 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:01.008735 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 05:42:01.008798 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":854}
I20260926 05:42:01.986901 31813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:01.986629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:59287)
0926 05:42:01.986769 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:01.986843 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260926 05:42:01.986913 31812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:01.986619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:41801)
0926 05:42:01.986768 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:01.986842 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260926 05:42:01.987022 31814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:01.986832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:50333)
0926 05:42:01.986922 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:01.986980 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":41,"threads_started":1}
W20260926 05:42:01.987088 31747 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:35903 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
W20260926 05:42:01.987089 31611 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:35903 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
W20260926 05:42:01.987217 31475 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:35903 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
I20260926 05:42:02.489529 31815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dccae96291a9413783a14543383a29ca)
I20260926 05:42:02.489728 31815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:02.489826 31816 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader dccae96291a9413783a14543383a29ca)
I20260926 05:42:02.489962 31816 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:02.490254 31816 leader_election.cc:288] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dccae96291a9413783a14543383a29ca (127.23.213.254:35903), 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253:39357)
I20260926 05:42:02.490307 31815 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers dccae96291a9413783a14543383a29ca (127.23.213.254:35903), b3f821498f104da68674c7532371e0b4 (127.23.213.252:40177)
I20260926 05:42:02.490571 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.490229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57750)
0926 05:42:02.490481 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.490532 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":233,"thread_start_us":45,"threads_started":1}
I20260926 05:42:02.490705 31819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.490362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57760)
0926 05:42:02.490573 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.490620 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":52,"threads_started":1}
W20260926 05:42:02.490859 31098 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
W20260926 05:42:02.490909 31280 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
I20260926 05:42:02.491436 31819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.491335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57768)
0926 05:42:02.491368 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.491403 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 05:42:02.491426 31817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.491340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57784)
0926 05:42:02.491377 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.491406 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 05:42:02.491524 31098 leader_election.cc:334] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dccae96291a9413783a14543383a29ca (127.23.213.254:35903): Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
W20260926 05:42:02.491529 31280 leader_election.cc:334] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer dccae96291a9413783a14543383a29ca (127.23.213.254:35903): Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
I20260926 05:42:02.493747 31820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.490399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:39357 (local address 127.0.0.1:42112)
0926 05:42:02.490649 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.490660 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:02.490721 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:02.491346 (+ 625us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:02.491347 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:02.491349 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:02.491408 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:02.491410 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.492153 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.492154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:02.492963 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:02.492964 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.493510 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.493511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:02.493576 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:02.493581 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:42:02.493634 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 05:42:02.493670 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"thread_start_us":56,"threads_started":1}
I20260926 05:42:02.493798 31822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.490554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42112 (local address 127.23.213.253:39357)
0926 05:42:02.491230 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:02.491232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:02.491239 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:42:02.491260 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:02.491261 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:02.491269 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:02.491340 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:02.491492 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.492077 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.493043 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.493438 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.493491 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:02.493497 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:02.493644 (+ 147us) server_negotiation.cc:299] Negotiation successful
0926 05:42:02.493683 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":40,"threads_started":1}
I20260926 05:42:02.493902 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.490234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:40177 (local address 127.0.0.1:40318)
0926 05:42:02.490678 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.490686 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:02.490731 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:02.491608 (+ 877us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:02.491610 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:02.491612 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:02.491644 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:02.491646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.492336 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.492336 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:02.493068 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:02.493069 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.493628 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.493629 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:02.493687 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:02.493690 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:02.493777 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:42:02.493821 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"thread_start_us":46,"threads_started":1}
I20260926 05:42:02.494045 31821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.490487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.23.213.252:40177)
0926 05:42:02.491009 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:02.491011 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:02.491019 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:02.491043 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:02.491044 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:02.491068 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:02.491158 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:02.491726 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.492260 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.493147 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:02.493528 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:02.493606 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:02.493614 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:02.493780 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 05:42:02.493820 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":37,"threads_started":1}
I20260926 05:42:02.494146 31121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f821498f104da68674c7532371e0b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fa2426091a94d768868fdf7ed313b13" is_pre_election: true
I20260926 05:42:02.494184 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fa2426091a94d768868fdf7ed313b13" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3f821498f104da68674c7532371e0b4" is_pre_election: true
I20260926 05:42:02.494251 31303 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4fa2426091a94d768868fdf7ed313b13 in term 1.
I20260926 05:42:02.494251 31121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f821498f104da68674c7532371e0b4 in term 1.
I20260926 05:42:02.494396 31097 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fa2426091a94d768868fdf7ed313b13, b3f821498f104da68674c7532371e0b4; no voters: dccae96291a9413783a14543383a29ca
I20260926 05:42:02.494396 31277 leader_election.cc:302] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fa2426091a94d768868fdf7ed313b13, b3f821498f104da68674c7532371e0b4; no voters: dccae96291a9413783a14543383a29ca
I20260926 05:42:02.494539 31815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 05:42:02.494591 31815 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dccae96291a9413783a14543383a29ca)
I20260926 05:42:02.494616 31815 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:42:02.494678 31816 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 05:42:02.494721 31816 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Starting leader election (detected failure of leader dccae96291a9413783a14543383a29ca)
I20260926 05:42:02.494750 31816 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:42:02.495291 31815 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:02.495422 31815 leader_election.cc:288] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [CANDIDATE]: Term 2 election: Requested vote from peers dccae96291a9413783a14543383a29ca (127.23.213.254:35903), b3f821498f104da68674c7532371e0b4 (127.23.213.252:40177)
I20260926 05:42:02.495400 31816 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:02.495592 31303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4fa2426091a94d768868fdf7ed313b13" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "b3f821498f104da68674c7532371e0b4"
I20260926 05:42:02.495640 31816 leader_election.cc:288] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 2 election: Requested vote from peers dccae96291a9413783a14543383a29ca (127.23.213.254:35903), 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253:39357)
I20260926 05:42:02.495682 31303 raft_consensus.cc:2379] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 4fa2426091a94d768868fdf7ed313b13 in current term 2: Already voted for candidate b3f821498f104da68674c7532371e0b4 in this term.
I20260926 05:42:02.495695 31820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.495585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57810)
0926 05:42:02.495619 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.495643 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 05:42:02.495764 31121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f821498f104da68674c7532371e0b4" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fa2426091a94d768868fdf7ed313b13"
I20260926 05:42:02.495850 31121 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate b3f821498f104da68674c7532371e0b4 in current term 2: Already voted for candidate 4fa2426091a94d768868fdf7ed313b13 in this term.
I20260926 05:42:02.495919 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.495489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57796)
0926 05:42:02.495587 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.495613 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260926 05:42:02.496044 31820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.495888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57818)
0926 05:42:02.495994 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.496021 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260926 05:42:02.496137 31280 leader_election.cc:334] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dccae96291a9413783a14543383a29ca (127.23.213.254:35903): Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
I20260926 05:42:02.496179 31280 leader_election.cc:302] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: b3f821498f104da68674c7532371e0b4; no voters: 4fa2426091a94d768868fdf7ed313b13, dccae96291a9413783a14543383a29ca
I20260926 05:42:02.496245 31816 raft_consensus.cc:2726] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 05:42:02.496254 31818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.496117 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:57830)
0926 05:42:02.496205 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.496228 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260926 05:42:02.496318 31098 leader_election.cc:334] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer dccae96291a9413783a14543383a29ca (127.23.213.254:35903): Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
I20260926 05:42:02.496357 31098 leader_election.cc:302] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4fa2426091a94d768868fdf7ed313b13; no voters: b3f821498f104da68674c7532371e0b4, dccae96291a9413783a14543383a29ca
I20260926 05:42:02.496443 31815 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 05:42:02.987596 31826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.987383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:37653)
0926 05:42:02.987508 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.987545 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":64,"threads_started":1}
I20260926 05:42:02.987596 31825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.987373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:46617)
0926 05:42:02.987519 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.987545 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260926 05:42:02.987874 31827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:02.987557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:41521)
0926 05:42:02.987810 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:02.987841 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":237,"thread_start_us":59,"threads_started":1}
I20260926 05:42:03.988310 31828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:03.988028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:46561)
0926 05:42:03.988209 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:03.988251 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":80,"threads_started":1}
I20260926 05:42:03.988310 31829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:03.988034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:52853)
0926 05:42:03.988209 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:03.988258 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
I20260926 05:42:03.988402 31830 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:03.988245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:48221)
0926 05:42:03.988333 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:03.988360 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74,"thread_start_us":39,"threads_started":1}
I20260926 05:42:04.534369 31831 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:04.534462 31831 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:04.534637 31831 leader_election.cc:288] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers dccae96291a9413783a14543383a29ca (127.23.213.254:35903), 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253:39357)
I20260926 05:42:04.534914 31121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f821498f104da68674c7532371e0b4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fa2426091a94d768868fdf7ed313b13" is_pre_election: true
I20260926 05:42:04.535003 31121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3f821498f104da68674c7532371e0b4 in term 2.
I20260926 05:42:04.534979 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.534741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:58096)
0926 05:42:04.534851 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.534938 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":56,"threads_started":1}
I20260926 05:42:04.535182 31277 leader_election.cc:302] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4fa2426091a94d768868fdf7ed313b13, b3f821498f104da68674c7532371e0b4; no voters:
I20260926 05:42:04.535286 31831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 05:42:04.535339 31831 raft_consensus.cc:482] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:04.535382 31831 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:42:04.535454 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.535339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:58102)
0926 05:42:04.535396 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.535429 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:42:04.535531 31280 leader_election.cc:334] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer dccae96291a9413783a14543383a29ca (127.23.213.254:35903): Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
I20260926 05:42:04.536190 31831 raft_consensus.cc:504] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:04.536389 31831 leader_election.cc:288] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 3 election: Requested vote from peers dccae96291a9413783a14543383a29ca (127.23.213.254:35903), 4fa2426091a94d768868fdf7ed313b13 (127.23.213.253:39357)
I20260926 05:42:04.536499 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.536390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:58114)
0926 05:42:04.536444 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.536475 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:42:04.536504 31121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "b3f821498f104da68674c7532371e0b4" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "4fa2426091a94d768868fdf7ed313b13"
I20260926 05:42:04.536612 31121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:42:04.536809 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.536725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:58120)
0926 05:42:04.536767 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.536790 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:42:04.536892 31280 leader_election.cc:334] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer dccae96291a9413783a14543383a29ca (127.23.213.254:35903): Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
I20260926 05:42:04.537274 31121 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3f821498f104da68674c7532371e0b4 in term 3.
I20260926 05:42:04.537469 31277 leader_election.cc:302] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4fa2426091a94d768868fdf7ed313b13, b3f821498f104da68674c7532371e0b4; no voters: dccae96291a9413783a14543383a29ca
I20260926 05:42:04.537603 31831 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:42:04.537765 31831 raft_consensus.cc:686] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [term 3 LEADER]: Becoming Leader. State: Replica: b3f821498f104da68674c7532371e0b4, State: Running, Role: LEADER
I20260926 05:42:04.537847 31831 consensus_queue.cc:237] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } }
I20260926 05:42:04.538219 31834 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f821498f104da68674c7532371e0b4. Latest consensus state: current_term: 3 leader_uuid: "b3f821498f104da68674c7532371e0b4" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:04.538321 31834 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:04.538491 31836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:42:04.538916 31836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:42:04.538972 31836 catalog_manager.cc:1259] Loaded cluster ID: b9443604a1104cce9a2871a783d8a847
I20260926 05:42:04.539023 31836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:42:04.539215 31836 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:42:04.539278 31836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Loaded TSK: 0
I20260926 05:42:04.539435 31836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:42:04.585940 31293 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:42:04.587592 31293 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:04.588769 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.588621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:58144)
0926 05:42:04.588697 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.588736 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:42:04.588774 31121 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [term 3 FOLLOWER]: Refusing update from remote peer b3f821498f104da68674c7532371e0b4: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 05:42:04.589054 31834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260926 05:42:04.589241 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.589116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.0.0.1:58152)
0926 05:42:04.589163 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.589214 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 05:42:04.589365 31280 consensus_peers.cc:487] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 -> Peer dccae96291a9413783a14543383a29ca (127.23.213.254:35903): Couldn't send request to peer dccae96291a9413783a14543383a29ca. Status: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 05:42:04.589826 31838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b3f821498f104da68674c7532371e0b4. Latest consensus state: current_term: 3 leader_uuid: "b3f821498f104da68674c7532371e0b4" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:04.590008 31838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:42:04.590505 31834 sys_catalog.cc:455] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "b3f821498f104da68674c7532371e0b4" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:04.590610 31834 sys_catalog.cc:458] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:04.590562 31838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "b3f821498f104da68674c7532371e0b4" committed_config { opid_index: 10 OBSOLETE_local: true peers { permanent_uuid: "dccae96291a9413783a14543383a29ca" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35903 } } peers { permanent_uuid: "4fa2426091a94d768868fdf7ed313b13" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 39357 } attrs { promote: false } } peers { permanent_uuid: "b3f821498f104da68674c7532371e0b4" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 40177 } attrs { promote: false } } }
I20260926 05:42:04.590647 31838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4fa2426091a94d768868fdf7ed313b13 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:42:04.590713 31839 mvcc.cc:205] Tried to move back new op lower bound from 7333483825513234432 to 7333483825307709440. Current Snapshot: MvccSnapshot[applied={T|T < 7333483825513234432 or (T in {7333483825513234432})}]
I20260926 05:42:04.595835 31843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.592617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38665 (local address 127.0.0.1:41240)
0926 05:42:04.592914 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.592923 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:04.592971 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:04.593283 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:04.593284 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:04.593286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:04.593343 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.593345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.594085 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.594086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.594821 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.594822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.595567 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.595568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.595636 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.595640 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:04.595720 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:42:04.595763 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":278,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260926 05:42:04.595836 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.592808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41240 (local address 127.23.213.195:38665)
0926 05:42:04.592958 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:04.593097 (+ 139us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:04.593107 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:04.593136 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:04.593137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:04.593150 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:04.593226 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:04.593430 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.593983 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.594912 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.595462 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.595559 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.595568 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:04.595727 (+ 159us) server_negotiation.cc:299] Negotiation successful
0926 05:42:04.595767 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":98,"threads_started":1}
I20260926 05:42:04.596462 31680 tablet_service.cc:1467] Processing CreateTablet for tablet 1938b53bc5e1427d994f49b7010f2dfc (DEFAULT_TABLE table=test-table [id=9ade26f1dc9146bab04770e5c3044b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:04.596612 31680 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1938b53bc5e1427d994f49b7010f2dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:04.596668 31832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.592552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:36013 (local address 127.0.0.1:57986)
0926 05:42:04.592632 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.592644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:04.592698 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:04.593184 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:04.593185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:04.593186 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:04.593241 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.593242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.594202 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.594203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.595451 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.595453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.596426 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.596427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.596487 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.596491 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:04.596558 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:42:04.596603 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 05:42:04.596791 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.592698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57986 (local address 127.23.213.194:36013)
0926 05:42:04.592973 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:04.592975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:04.592986 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:42:04.593023 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:04.593024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:04.593038 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:04.593124 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:04.593376 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.594094 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.595570 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.596225 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.596363 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.596374 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:04.596653 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 05:42:04.596705 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":57,"threads_started":1}
I20260926 05:42:04.597004 31544 tablet_service.cc:1467] Processing CreateTablet for tablet 1938b53bc5e1427d994f49b7010f2dfc (DEFAULT_TABLE table=test-table [id=9ade26f1dc9146bab04770e5c3044b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:04.597112 31544 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1938b53bc5e1427d994f49b7010f2dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:04.597497 31844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.592657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:33691 (local address 127.0.0.1:52678)
0926 05:42:04.593210 (+ 553us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.593222 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:04.593279 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:04.594009 (+ 730us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:04.594010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:04.594011 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:04.594054 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.594056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.595120 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.595121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.596322 (+ 1201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.596326 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.597180 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.597181 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.597253 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.597257 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:04.597373 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 05:42:04.597418 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":97,"threads_started":1}
I20260926 05:42:04.597571 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.592787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52678 (local address 127.23.213.193:33691)
0926 05:42:04.592980 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:04.592982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:04.593817 (+ 835us) server_negotiation.cc:407] Connection header received
0926 05:42:04.593865 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:04.593867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:04.593880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:04.593956 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:04.594143 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.595006 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.596433 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.597058 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.597181 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.597198 (+ 17us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:04.597384 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 05:42:04.597472 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":78,"threads_started":1}
I20260926 05:42:04.597860 31408 tablet_service.cc:1467] Processing CreateTablet for tablet 1938b53bc5e1427d994f49b7010f2dfc (DEFAULT_TABLE table=test-table [id=9ade26f1dc9146bab04770e5c3044b72]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:04.597990 31408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1938b53bc5e1427d994f49b7010f2dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:04.598268 31848 tablet_bootstrap.cc:492] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: Bootstrap starting.
I20260926 05:42:04.598762 31848 tablet_bootstrap.cc:654] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:04.598971 31849 tablet_bootstrap.cc:492] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: Bootstrap starting.
I20260926 05:42:04.599366 31849 tablet_bootstrap.cc:654] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:04.599589 31848 tablet_bootstrap.cc:492] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: No bootstrap required, opened a new log
I20260926 05:42:04.599720 31848 ts_tablet_manager.cc:1397] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:04.599913 31849 tablet_bootstrap.cc:492] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: No bootstrap required, opened a new log
I20260926 05:42:04.599974 31849 ts_tablet_manager.cc:1397] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:04.600052 31848 raft_consensus.cc:348] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.600106 31849 raft_consensus.cc:348] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.600153 31848 raft_consensus.cc:374] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:04.600173 31849 raft_consensus.cc:374] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:04.600174 31848 raft_consensus.cc:729] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31bf19fb701148f58f69c8b548f2cbc5, State: Initialized, Role: FOLLOWER
I20260926 05:42:04.600198 31849 raft_consensus.cc:729] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 327e803d0d744eb8933eff3fc78e4d2d, State: Initialized, Role: FOLLOWER
I20260926 05:42:04.600253 31849 consensus_queue.cc:260] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.600274 31848 consensus_queue.cc:260] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.600399 31849 ts_tablet_manager.cc:1428] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:04.600448 31848 ts_tablet_manager.cc:1428] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:42:04.600605 31852 tablet_bootstrap.cc:492] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a: Bootstrap starting.
I20260926 05:42:04.601003 31852 tablet_bootstrap.cc:654] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:04.601020 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.600752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:36421)
0926 05:42:04.600920 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.600973 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260926 05:42:04.601140 31746 heartbeater.cc:503] Master 127.23.213.252:40177 was elected leader, sending a full tablet report...
I20260926 05:42:04.601343 31610 heartbeater.cc:503] Master 127.23.213.252:40177 was elected leader, sending a full tablet report...
I20260926 05:42:04.601900 31852 tablet_bootstrap.cc:492] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a: No bootstrap required, opened a new log
I20260926 05:42:04.601985 31852 ts_tablet_manager.cc:1397] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:04.602038 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.600990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:53721)
0926 05:42:04.601963 (+ 973us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.602002 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":949,"thread_start_us":90,"threads_started":1}
I20260926 05:42:04.602169 31852 raft_consensus.cc:348] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.602244 31852 raft_consensus.cc:374] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:04.602259 31852 raft_consensus.cc:729] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2add9c82ce14ffeb3db41d11218936a, State: Initialized, Role: FOLLOWER
I20260926 05:42:04.602316 31852 consensus_queue.cc:260] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.602492 31852 ts_tablet_manager.cc:1428] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:04.602881 31859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.602670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:51799)
0926 05:42:04.602799 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.602836 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":99,"threads_started":1}
I20260926 05:42:04.603118 31474 heartbeater.cc:503] Master 127.23.213.252:40177 was elected leader, sending a full tablet report...
I20260926 05:42:04.603246 31859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.603088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:36001)
0926 05:42:04.603195 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.603218 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":69,"threads_started":1}
I20260926 05:42:04.681243 31853 raft_consensus.cc:482] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:04.681380 31853 raft_consensus.cc:504] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.681819 31853 leader_election.cc:288] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195:38665), e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691)
I20260926 05:42:04.685119 31847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.682107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:51209 (local address 127.23.213.195:38665)
0926 05:42:04.682150 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:04.682152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:04.682271 (+ 119us) server_negotiation.cc:407] Connection header received
0926 05:42:04.682338 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:04.682339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:04.682348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:04.682409 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:04.682573 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.683145 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.684376 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.684726 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.684816 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.684827 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:04.685005 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 05:42:04.685047 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:42:04.685119 31846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.682058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:60679 (local address 127.23.213.193:33691)
0926 05:42:04.682114 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:04.682116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:04.682129 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:42:04.682176 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:04.682177 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:04.682190 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:04.682277 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:04.682444 (+ 167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.683139 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.684278 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.684724 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.684824 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.684834 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:04.684974 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 05:42:04.685023 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:42:04.685120 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.681980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:33691 (local address 127.23.213.194:60679)
0926 05:42:04.682032 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.682055 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:04.682118 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:04.682288 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:04.682289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:04.682292 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:04.682357 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.682359 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.683277 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.683278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.684186 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.684189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.684823 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.684824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.684893 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.684897 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:04.684939 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 05:42:04.684997 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":33}
I20260926 05:42:04.685218 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.681995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38665 (local address 127.23.213.194:51209)
0926 05:42:04.682188 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.682199 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:04.682263 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:04.682445 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:04.682446 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:04.682448 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:04.682494 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.682496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.683235 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.683236 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.684281 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:04.684284 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.684819 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.684820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:04.684900 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.684903 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:04.684992 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:42:04.685034 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":46,"thread_start_us":117,"threads_started":1}
I20260926 05:42:04.685451 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1938b53bc5e1427d994f49b7010f2dfc" candidate_uuid: "327e803d0d744eb8933eff3fc78e4d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2add9c82ce14ffeb3db41d11218936a" is_pre_election: true
I20260926 05:42:04.685572 31428 raft_consensus.cc:2454] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327e803d0d744eb8933eff3fc78e4d2d in term 0.
I20260926 05:42:04.685513 31700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1938b53bc5e1427d994f49b7010f2dfc" candidate_uuid: "327e803d0d744eb8933eff3fc78e4d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31bf19fb701148f58f69c8b548f2cbc5" is_pre_election: true
I20260926 05:42:04.685662 31700 raft_consensus.cc:2454] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 327e803d0d744eb8933eff3fc78e4d2d in term 0.
I20260926 05:42:04.685804 31498 leader_election.cc:302] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327e803d0d744eb8933eff3fc78e4d2d, e2add9c82ce14ffeb3db41d11218936a; no voters:
I20260926 05:42:04.686017 31853 raft_consensus.cc:2781] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:04.686079 31853 raft_consensus.cc:482] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:04.686097 31853 raft_consensus.cc:3037] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:04.686879 31853 raft_consensus.cc:504] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.687050 31853 leader_election.cc:288] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [CANDIDATE]: Term 1 election: Requested vote from peers 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195:38665), e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691)
I20260926 05:42:04.687232 31428 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1938b53bc5e1427d994f49b7010f2dfc" candidate_uuid: "327e803d0d744eb8933eff3fc78e4d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e2add9c82ce14ffeb3db41d11218936a"
I20260926 05:42:04.687230 31700 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1938b53bc5e1427d994f49b7010f2dfc" candidate_uuid: "327e803d0d744eb8933eff3fc78e4d2d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31bf19fb701148f58f69c8b548f2cbc5"
I20260926 05:42:04.687321 31428 raft_consensus.cc:3037] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:04.687321 31700 raft_consensus.cc:3037] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:04.688148 31428 raft_consensus.cc:2454] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327e803d0d744eb8933eff3fc78e4d2d in term 1.
I20260926 05:42:04.688242 31700 raft_consensus.cc:2454] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 327e803d0d744eb8933eff3fc78e4d2d in term 1.
I20260926 05:42:04.688329 31498 leader_election.cc:302] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 327e803d0d744eb8933eff3fc78e4d2d, e2add9c82ce14ffeb3db41d11218936a; no voters:
I20260926 05:42:04.688446 31853 raft_consensus.cc:2781] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:04.688640 31853 raft_consensus.cc:686] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 1 LEADER]: Becoming Leader. State: Replica: 327e803d0d744eb8933eff3fc78e4d2d, State: Running, Role: LEADER
I20260926 05:42:04.688730 31853 consensus_queue.cc:237] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } }
I20260926 05:42:04.689118 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.688890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:33887)
0926 05:42:04.689041 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.689084 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129}
I20260926 05:42:04.689465 31293 catalog_manager.cc:5495] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d reported cstate change: term changed from 0 to 1, leader changed from <none> to 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194). New cstate: current_term: 1 leader_uuid: "327e803d0d744eb8933eff3fc78e4d2d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "327e803d0d744eb8933eff3fc78e4d2d" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36013 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 } health_report { overall_health: UNKNOWN } } }
I20260926 05:42:04.714869 31845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.711726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58002 (local address 127.23.213.194:36013)
0926 05:42:04.711799 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:04.711806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:04.711813 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:42:04.711849 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:04.711849 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:04.711867 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:04.711948 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:04.712105 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.712584 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.713675 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:04.714071 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:04.714159 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:04.714393 (+ 234us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:04.714535 (+ 142us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:04.714614 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:04.714721 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 05:42:04.714770 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:42:04.716526 31428 raft_consensus.cc:1261] T 1938b53bc5e1427d994f49b7010f2dfc P e2add9c82ce14ffeb3db41d11218936a [term 1 FOLLOWER]: Refusing update from remote peer 327e803d0d744eb8933eff3fc78e4d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:04.716600 31700 raft_consensus.cc:1261] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 FOLLOWER]: Refusing update from remote peer 327e803d0d744eb8933eff3fc78e4d2d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:04.716850 31862 consensus_queue.cc:1035] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e2add9c82ce14ffeb3db41d11218936a" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 33691 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:04.716967 31853 consensus_queue.cc:1035] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [LEADER]: Connected to new peer: Peer: permanent_uuid: "31bf19fb701148f58f69c8b548f2cbc5" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38665 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:04.717841 31859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.717658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:47489)
0926 05:42:04.717723 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.717795 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 05:42:04.718165 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.717999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:42265)
0926 05:42:04.718103 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.718132 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260926 05:42:04.719208 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.719002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:47101)
0926 05:42:04.719138 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.719173 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260926 05:42:04.719224 31868 mvcc.cc:205] Tried to move back new op lower bound from 7333483826036117504 to 7333483825925500928. Current Snapshot: MvccSnapshot[applied={T|T < 7333483826036117504}]
I20260926 05:42:04.719624 31855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.719464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.195:52225)
0926 05:42:04.719572 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.719600 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260926 05:42:04.721166 31859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.721070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.193:53333)
0926 05:42:04.721108 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.721130 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 05:42:04.721369 31861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:04.721136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35903 (local address 127.23.213.194:42983)
0926 05:42:04.721312 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.721342 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:35903: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260926 05:42:04.722086 31293 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:04.722190 31293 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40302:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:04.723486 31293 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b3f821498f104da68674c7532371e0b4: Sending DeleteTablet for 3 replicas of tablet 1938b53bc5e1427d994f49b7010f2dfc
I20260926 05:42:04.723793 31680 tablet_service.cc:1514] Processing DeleteTablet for tablet 1938b53bc5e1427d994f49b7010f2dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:04 UTC) from {username='oryx', principal='oryx/127.23.213.252@KRBTEST.COM'} at 127.0.0.1:41240
I20260926 05:42:04.723934 31544 tablet_service.cc:1514] Processing DeleteTablet for tablet 1938b53bc5e1427d994f49b7010f2dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:04 UTC) from {username='oryx', principal='oryx/127.23.213.252@KRBTEST.COM'} at 127.0.0.1:57986
I20260926 05:42:04.724126 31875 tablet_replica.cc:330] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: stopping tablet replica
I20260926 05:42:04.724188 31875 raft_consensus.cc:2229] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:42:04.724217 31408 tablet_service.cc:1514] Processing DeleteTablet for tablet 1938b53bc5e1427d994f49b7010f2dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:04 UTC) from {username='oryx', principal='oryx/127.23.213.252@KRBTEST.COM'} at 127.0.0.1:52678
I20260926 05:42:04.724270 31875 raft_consensus.cc:2258] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:04.724542 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 31345
I20260926 05:42:04.724583 31875 ts_tablet_manager.cc:1905] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:04.726773 31874 tablet_replica.cc:330] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: stopping tablet replica
I20260926 05:42:04.726882 31874 raft_consensus.cc:2229] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:04.726898 31875 ts_tablet_manager.cc:1918] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:04.726960 31875 log.cc:1196] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/1938b53bc5e1427d994f49b7010f2dfc
I20260926 05:42:04.727048 31874 raft_consensus.cc:2258] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:04.727214 31875 ts_tablet_manager.cc:1939] T 1938b53bc5e1427d994f49b7010f2dfc P 327e803d0d744eb8933eff3fc78e4d2d: Deleting consensus metadata
I20260926 05:42:04.727427 31874 ts_tablet_manager.cc:1905] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:04.728132 31278 catalog_manager.cc:4841] TS 327e803d0d744eb8933eff3fc78e4d2d (127.23.213.194:36013): tablet 1938b53bc5e1427d994f49b7010f2dfc (table test-table [id=9ade26f1dc9146bab04770e5c3044b72]) successfully deleted
I20260926 05:42:04.729271 31874 ts_tablet_manager.cc:1918] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:04.729336 31874 log.cc:1196] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/1938b53bc5e1427d994f49b7010f2dfc
I20260926 05:42:04.729592 31874 ts_tablet_manager.cc:1939] T 1938b53bc5e1427d994f49b7010f2dfc P 31bf19fb701148f58f69c8b548f2cbc5: Deleting consensus metadata
I20260926 05:42:04.729945 31279 catalog_manager.cc:4841] TS 31bf19fb701148f58f69c8b548f2cbc5 (127.23.213.195:38665): tablet 1938b53bc5e1427d994f49b7010f2dfc (table test-table [id=9ade26f1dc9146bab04770e5c3044b72]) successfully deleted
W20260926 05:42:04.732085 31844 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:04.731953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:33691 (local address 127.0.0.1:52694)
0926 05:42:04.732003 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:04.732031 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:33691: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:42:04.732198 31278 catalog_manager.cc:4584] TS e2add9c82ce14ffeb3db41d11218936a (127.23.213.193:33691): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1938b53bc5e1427d994f49b7010f2dfc: Network error: Client connection negotiation failed: client connection to 127.23.213.193:33691: connect: Connection reset by peer (error 104)
I20260926 05:42:04.732262 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 31481
I20260926 05:42:04.737778 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 31617
I20260926 05:42:04.743268 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 30974
I20260926 05:42:04.747720 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 31147
2026-09-26T05:42:04Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (7030 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:42:04 dist-test-slave-1fdt krb5kdc[31879](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:42:04 dist-test-slave-1fdt krb5kdc[31879](info): set up 2 sockets
Sep 26 05:42:04 dist-test-slave-1fdt krb5kdc[31879](info): commencing operation
krb5kdc: starting...
W20260926 05:42:05.790201 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:42:05 dist-test-slave-1fdt krb5kdc[31879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:42:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:42:05Z Disabled control of system clock
WARNING: no policy specified for oryx/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab.
Entry for principal oryx/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab.
I20260926 05:42:05.908877 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:46775
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab
--principal=oryx/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:46775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:05.989897 31892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:05.990141 31892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:05.990180 31892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:05.991323 31892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:05.991396 31892 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 05:42:05.991418 31892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:05.991441 31892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:05.991469 31892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:05.991493 31892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:05.992986 31892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:46775
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.254.keytab
--principal=oryx/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:46775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:05.993263 31892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:05.993490 31892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:05.998512 31897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:05.998647 31892 server_base.cc:1034] running on GCE node
W20260926 05:42:05.998488 31898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:05.998473 31900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:05.998984 31892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:05.999301 31892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:06.000442 31892 hybrid_clock.cc:648] HybridClock initialized: now 1790401326000433 us; error 32 us; skew 500 ppm
Sep 26 05:42:05 dist-test-slave-1fdt krb5kdc[31879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401325, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:06.002955 31892 init.cc:377] Logged in from keytab as oryx/127.23.213.254@KRBTEST.COM (short username oryx)
I20260926 05:42:06.003388 31892 webserver.cc:460] Webserver started at http://127.23.213.254:41729/ using document root <none> and password file <none>
I20260926 05:42:06.003629 31892 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:06.003697 31892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:06.003841 31892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:06.004915 31892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "79a9cee23f1047019a5cb8a9b326d73b"
format_stamp: "Formatted at 2026-09-26 05:42:05 on dist-test-slave-1fdt"
server_key: "47588f83882f63c3800b069a04e21615"
server_key_iv: "c13c80b6c69ef9e57c0cdc3640b76d7b"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.005268 31892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "79a9cee23f1047019a5cb8a9b326d73b"
format_stamp: "Formatted at 2026-09-26 05:42:05 on dist-test-slave-1fdt"
server_key: "47588f83882f63c3800b069a04e21615"
server_key_iv: "c13c80b6c69ef9e57c0cdc3640b76d7b"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.006698 31892 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:42:06.007476 31907 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.007717 31892 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:42:06.007825 31892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "79a9cee23f1047019a5cb8a9b326d73b"
format_stamp: "Formatted at 2026-09-26 05:42:05 on dist-test-slave-1fdt"
server_key: "47588f83882f63c3800b069a04e21615"
server_key_iv: "c13c80b6c69ef9e57c0cdc3640b76d7b"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.007898 31892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:06.026428 31892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:06.028684 31892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:06.028892 31892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:06.033887 31892 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:46775
I20260926 05:42:06.033983 31969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:46775 every 8 connection(s)
I20260926 05:42:06.034334 31892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:42:06.035168 31970 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:06.037906 31970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Bootstrap starting.
I20260926 05:42:06.038555 31970 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:06.038862 31970 log.cc:824] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:06.039646 31970 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: No bootstrap required, opened a new log
I20260926 05:42:06.042052 31970 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a9cee23f1047019a5cb8a9b326d73b" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46775 } }
I20260926 05:42:06.042204 31970 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:06.042229 31970 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79a9cee23f1047019a5cb8a9b326d73b, State: Initialized, Role: FOLLOWER
I20260926 05:42:06.042384 31970 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a9cee23f1047019a5cb8a9b326d73b" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46775 } }
I20260926 05:42:06.042478 31970 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:42:06.042503 31970 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:42:06.042536 31970 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:06.043094 31970 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a9cee23f1047019a5cb8a9b326d73b" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46775 } }
I20260926 05:42:06.043191 31970 leader_election.cc:302] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79a9cee23f1047019a5cb8a9b326d73b; no voters:
I20260926 05:42:06.043390 31970 leader_election.cc:288] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:42:06.043426 31973 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:06.043592 31973 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [term 1 LEADER]: Becoming Leader. State: Replica: 79a9cee23f1047019a5cb8a9b326d73b, State: Running, Role: LEADER
I20260926 05:42:06.043856 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 31892
I20260926 05:42:06.043941 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:42:06.044092 24407 external_mini_cluster.cc:1428] Setting key 6d72a5a9a20549e9aa212cb02ec83c3f
I20260926 05:42:06.043716 31970 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:06.044422 31973 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a9cee23f1047019a5cb8a9b326d73b" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46775 } }
I20260926 05:42:06.045233 31974 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79a9cee23f1047019a5cb8a9b326d73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a9cee23f1047019a5cb8a9b326d73b" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46775 } } }
I20260926 05:42:06.045325 31974 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:06.045607 31975 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79a9cee23f1047019a5cb8a9b326d73b. Latest consensus state: current_term: 1 leader_uuid: "79a9cee23f1047019a5cb8a9b326d73b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79a9cee23f1047019a5cb8a9b326d73b" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46775 } } }
I20260926 05:42:06.045703 31975 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b [sys.catalog]: This master's current role is: LEADER
W20260926 05:42:06.046362 31991 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:42:06.046435 31991 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:42:06.046622 31986 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:42:06.046811 31986 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:42:06.048532 31986 catalog_manager.cc:1347] Generated new cluster ID: de4c4a576b654f998b6f1dc2146df8ad
I20260926 05:42:06.048593 31986 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401325, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:06.057128 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.044569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51116 (local address 127.23.213.254:46775)
0926 05:42:06.044757 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.044768 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.045081 (+ 313us) server_negotiation.cc:407] Connection header received
0926 05:42:06.045320 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.045329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.045395 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.045577 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.046670 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.048226 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.049451 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.050151 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.050262 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.053225 (+ 2963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.053229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.053229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.053244 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.055371 (+ 2127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.055793 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.055797 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.055797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.055910 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.056124 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.056125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.056125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.056454 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.056624 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.056723 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.056868 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":78,"threads_started":1}
I20260926 05:42:06.080004 31986 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:42:06.080657 31986 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:42:06.085985 31986 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Generated new TSK 0
I20260926 05:42:06.086202 31986 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab.
Entry for principal oryx/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab.
I20260926 05:42:06.113207 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46775
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab
--principal=oryx/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:06.200790 31996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:06.200999 31996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:06.201025 31996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:06.202256 31996 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:06.202311 31996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:06.202333 31996 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:42:06.202384 31996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:42:06.203975 31996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab
--principal=oryx/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31996
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:06.204241 31996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:06.204491 31996 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:06.209569 32004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:06.209590 32001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:06.209792 31996 server_base.cc:1034] running on GCE node
W20260926 05:42:06.209622 32002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:06.210179 31996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:06.210434 31996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:06.211684 31996 hybrid_clock.cc:648] HybridClock initialized: now 1790401326211664 us; error 44 us; skew 500 ppm
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:06.214304 31996 init.cc:377] Logged in from keytab as oryx/127.23.213.193@KRBTEST.COM (short username oryx)
I20260926 05:42:06.214749 31996 webserver.cc:460] Webserver started at http://127.23.213.193:37649/ using document root <none> and password file <none>
I20260926 05:42:06.215029 31996 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:06.215078 31996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:06.215250 31996 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:06.216332 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "4128951d0e8e4051ac1597eda78a3747"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "8d8ed57469575ba8ce23dc61e673b964"
server_key_iv: "fecf4753e7050fb6fce9d3412714fd20"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.216688 31996 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "4128951d0e8e4051ac1597eda78a3747"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "8d8ed57469575ba8ce23dc61e673b964"
server_key_iv: "fecf4753e7050fb6fce9d3412714fd20"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.218034 31996 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:42:06.218734 32011 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.218945 31996 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:06.219009 31996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "4128951d0e8e4051ac1597eda78a3747"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "8d8ed57469575ba8ce23dc61e673b964"
server_key_iv: "fecf4753e7050fb6fce9d3412714fd20"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.219099 31996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:06.231154 31996 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:06.233258 31996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:06.233431 31996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:06.234057 31996 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:06.234128 31996 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.234179 31996 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:06.234210 31996 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:06.242314 31996 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:46813
I20260926 05:42:06.242405 32126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:46813 every 8 connection(s)
I20260926 05:42:06.242787 31996 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:42:06.244837 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.234513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:56735 (local address 127.23.213.254:46775)
0926 05:42:06.234585 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.234587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.234783 (+ 196us) server_negotiation.cc:407] Connection header received
0926 05:42:06.235284 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.235286 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.235304 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.235401 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.236174 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.237133 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.238374 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.238793 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.238891 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.241567 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.241568 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.241569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.241579 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.243419 (+ 1840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.243849 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.243850 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.243851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.243911 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.244176 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.244177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.244178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.244338 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.244406 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.244702 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.244773 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:42:06.245076 32020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.234499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46775 (local address 127.23.213.193:56735)
0926 05:42:06.234648 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.234687 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.235184 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.235450 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.235454 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.235744 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:06.236042 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.236049 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.237234 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.237235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.238299 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.238302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.238911 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.238913 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.239087 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.239616 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:06.239625 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:06.241446 (+ 1821us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:06.243516 (+ 2070us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.243522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.243524 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.243767 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.243997 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.243997 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.243998 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.244106 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.244406 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:06.244415 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:06.244573 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.244677 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.244817 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":79,"threads_started":1}
I20260926 05:42:06.245617 32127 heartbeater.cc:347] Connected to a master server at 127.23.213.254:46775
I20260926 05:42:06.245699 32127 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:06.245901 32127 heartbeater.cc:511] Master 127.23.213.254:46775 requested a full tablet report, sending...
I20260926 05:42:06.246482 31923 ts_manager.cc:194] Registered new tserver with Master: 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813)
I20260926 05:42:06.247226 31923 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:56735
I20260926 05:42:06.248741 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 31996
I20260926 05:42:06.248849 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:42:06.249025 24407 external_mini_cluster.cc:1428] Setting key a7a4ff5e437d7182e409f64bcc59934e
I20260926 05:42:06.253566 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.246823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:46137 (local address 127.23.213.254:46775)
0926 05:42:06.246870 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.246871 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.247489 (+ 618us) server_negotiation.cc:407] Connection header received
0926 05:42:06.247563 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.247564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.247573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.247627 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.247945 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.248412 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.249449 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.249640 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.249726 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.250813 (+ 1087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.250814 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.250814 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.250821 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.252499 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.252888 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.252889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.252889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.252944 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.253144 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.253144 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.253145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.253239 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.253287 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.253444 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.253504 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:42:06.253716 32133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.246742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46775 (local address 127.23.213.193:46137)
0926 05:42:06.247413 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.247423 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.247502 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.247632 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.247632 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.247808 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:06.247861 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.247863 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.248506 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.248507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.249369 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.249372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.249737 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.249738 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.249854 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.250298 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:06.250304 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:06.250725 (+ 421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:06.252590 (+ 1865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.252591 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.252592 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.252814 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.253010 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.253011 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.253011 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.253084 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.253299 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:06.253300 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:06.253351 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.253428 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.253564 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":62,"threads_started":1}
WARNING: no policy specified for oryx/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab.
Entry for principal oryx/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab.
I20260926 05:42:06.274823 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46775
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab
--principal=oryx/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:06.356357 32136 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:06.356571 32136 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:06.356596 32136 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:06.357692 32136 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:06.357724 32136 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:06.357777 32136 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:42:06.357795 32136 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:42:06.359331 32136 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab
--principal=oryx/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32136
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:06.359527 32136 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:06.359709 32136 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:06.364822 32141 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:06.364837 32144 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:06.364885 32136 server_base.cc:1034] running on GCE node
W20260926 05:42:06.364840 32142 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:06.365365 32136 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:06.365662 32136 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:06.366825 32136 hybrid_clock.cc:648] HybridClock initialized: now 1790401326366825 us; error 46 us; skew 500 ppm
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:06.369345 32136 init.cc:377] Logged in from keytab as oryx/127.23.213.194@KRBTEST.COM (short username oryx)
I20260926 05:42:06.369877 32136 webserver.cc:460] Webserver started at http://127.23.213.194:36245/ using document root <none> and password file <none>
I20260926 05:42:06.370086 32136 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:06.370126 32136 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:06.370216 32136 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:06.371204 32136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "fea52a05957b41598459e22945bfab43"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "1a553952eafe35e496309ec58eb26137"
server_key_iv: "01a80296ea679a4eec693e5961f90f46"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.371495 32136 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "fea52a05957b41598459e22945bfab43"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "1a553952eafe35e496309ec58eb26137"
server_key_iv: "01a80296ea679a4eec693e5961f90f46"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.373152 32136 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 05:42:06.373945 32151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.374192 32136 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:06.374295 32136 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "fea52a05957b41598459e22945bfab43"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "1a553952eafe35e496309ec58eb26137"
server_key_iv: "01a80296ea679a4eec693e5961f90f46"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.374385 32136 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:06.382769 32136 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:06.385097 32136 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:06.385295 32136 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:06.385934 32136 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:06.385983 32136 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.386034 32136 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:06.386060 32136 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:06.393335 32136 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:45781
I20260926 05:42:06.393401 32266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:45781 every 8 connection(s)
I20260926 05:42:06.393765 32136 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:42:06.396239 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.386273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:37597 (local address 127.23.213.254:46775)
0926 05:42:06.386331 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.386334 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.386612 (+ 278us) server_negotiation.cc:407] Connection header received
0926 05:42:06.387391 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.387392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.387408 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.387491 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.388422 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.389008 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.390214 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.390441 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.390532 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.393218 (+ 2686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.393220 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.393220 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.393231 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.394777 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.395247 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.395248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.395249 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.395313 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.395572 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.395573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.395573 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.395700 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.395771 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.396067 (+ 296us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.396149 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:42:06.396433 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.386282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46775 (local address 127.23.213.194:37597)
0926 05:42:06.386444 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.386503 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.387225 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.387586 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.387591 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.387996 (+ 405us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:06.388300 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.388308 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.389111 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.389112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.390119 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.390122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.390557 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.390558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.390684 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.391214 (+ 530us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:06.391227 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:06.393065 (+ 1838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:06.394917 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.394924 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.394925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.395148 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.395388 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.395389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.395390 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.395501 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.395773 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:06.395780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:06.395929 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.396031 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.396192 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":88,"threads_started":1}
I20260926 05:42:06.396904 32267 heartbeater.cc:347] Connected to a master server at 127.23.213.254:46775
I20260926 05:42:06.397010 32267 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:06.397214 32267 heartbeater.cc:511] Master 127.23.213.254:46775 requested a full tablet report, sending...
I20260926 05:42:06.397601 31924 ts_manager.cc:194] Registered new tserver with Master: fea52a05957b41598459e22945bfab43 (127.23.213.194:45781)
I20260926 05:42:06.398118 31924 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:37597
I20260926 05:42:06.399533 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 32136
I20260926 05:42:06.399639 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:42:06.399818 24407 external_mini_cluster.cc:1428] Setting key 307f1378c0d41fcebc1ab4efa4984b1d
I20260926 05:42:06.403975 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.398051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:53063 (local address 127.23.213.254:46775)
0926 05:42:06.398097 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.398099 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.398224 (+ 125us) server_negotiation.cc:407] Connection header received
0926 05:42:06.398293 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.398294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.398304 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.398376 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.398669 (+ 293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.399141 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.400151 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.400360 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.400438 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.401348 (+ 910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.401349 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.401350 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.401356 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.402754 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.403123 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.403124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.403124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.403176 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.403390 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.403391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.403391 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.403539 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.403644 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.403801 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.403882 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:42:06.403990 32273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.397992 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46775 (local address 127.23.213.194:53063)
0926 05:42:06.398149 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.398166 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.398237 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.398381 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.398382 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.398528 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:06.398590 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.398592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.399232 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.399233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.400064 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.400067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.400448 (+ 381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.400449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.400519 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.400965 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:06.400970 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:06.401244 (+ 274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:06.402857 (+ 1613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.402858 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.402858 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.403034 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.403252 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.403253 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.403253 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.403311 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.403640 (+ 329us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:06.403641 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:06.403688 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.403776 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.403888 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for oryx/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab.
Entry for principal oryx/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab.
I20260926 05:42:06.425668 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46775
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab
--principal=oryx/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:06.507716 32276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:06.507925 32276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:06.507949 32276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:06.509055 32276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:06.509092 32276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:06.509112 32276 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 05:42:06.509130 32276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:42:06.510718 32276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab
--principal=oryx/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.23.213.254:46775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:06.510942 32276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:06.511129 32276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:06.516276 32284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:06.516372 32281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:06.516276 32282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:06.516564 32276 server_base.cc:1034] running on GCE node
I20260926 05:42:06.516804 32276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:06.517041 32276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:06.518181 32276 hybrid_clock.cc:648] HybridClock initialized: now 1790401326518164 us; error 35 us; skew 500 ppm
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:06.520700 32276 init.cc:377] Logged in from keytab as oryx/127.23.213.195@KRBTEST.COM (short username oryx)
I20260926 05:42:06.521210 32276 webserver.cc:460] Webserver started at http://127.23.213.195:43961/ using document root <none> and password file <none>
I20260926 05:42:06.521435 32276 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:06.521479 32276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:06.521638 32276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:06.522841 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "4cddb156244a4ab6b6833fa673046b05"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "b40535f90de2a32724bc241eab69df59"
server_key_iv: "faea54780a7e042f673063ae7077698e"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.523196 32276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "4cddb156244a4ab6b6833fa673046b05"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "b40535f90de2a32724bc241eab69df59"
server_key_iv: "faea54780a7e042f673063ae7077698e"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.524518 32276 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:42:06.525233 32291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.525413 32276 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:42:06.525501 32276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "4cddb156244a4ab6b6833fa673046b05"
format_stamp: "Formatted at 2026-09-26 05:42:06 on dist-test-slave-1fdt"
server_key: "b40535f90de2a32724bc241eab69df59"
server_key_iv: "faea54780a7e042f673063ae7077698e"
server_key_version: "encryptionkey@0"
I20260926 05:42:06.525584 32276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:06.563687 32276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:06.565686 32276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:06.565915 32276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:06.566471 32276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:06.566509 32276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.566533 32276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:06.566596 32276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:06.573518 32276 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:38611
I20260926 05:42:06.573591 32406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:38611 every 8 connection(s)
I20260926 05:42:06.573951 32276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:42:06.576478 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.567119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:54277 (local address 127.23.213.254:46775)
0926 05:42:06.567177 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.567180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.567434 (+ 254us) server_negotiation.cc:407] Connection header received
0926 05:42:06.568090 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.568091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.568108 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.568191 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.569147 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.570028 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.571230 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.571562 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.571638 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.574029 (+ 2391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.574030 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.574031 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.574039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.575325 (+ 1286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.575699 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.575700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.575700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.575749 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.575958 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.575958 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.575959 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.576067 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.576116 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.576355 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.576415 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:42:06.576705 32300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.567140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46775 (local address 127.23.213.195:54277)
0926 05:42:06.567297 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.567334 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.567931 (+ 597us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.568274 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.568278 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.568673 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:06.569026 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.569034 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.570150 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.570151 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.571128 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.571131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.571662 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.571663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.571774 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.572365 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:06.572382 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:06.573931 (+ 1549us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:06.575408 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.575413 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.575415 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.575632 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.575807 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.575807 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.575807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.575904 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.576123 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:06.576127 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:06.576239 (+ 112us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.576320 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.576452 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260926 05:42:06.577219 32407 heartbeater.cc:347] Connected to a master server at 127.23.213.254:46775
I20260926 05:42:06.577301 32407 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:06.577483 32407 heartbeater.cc:511] Master 127.23.213.254:46775 requested a full tablet report, sending...
I20260926 05:42:06.577889 31923 ts_manager.cc:194] Registered new tserver with Master: 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611)
I20260926 05:42:06.578348 31923 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:54277
I20260926 05:42:06.581190 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 32276
I20260926 05:42:06.581293 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:42:06.581494 24407 external_mini_cluster.cc:1428] Setting key 9e2f1fd327c8890d0e960e348143f573
I20260926 05:42:06.582461 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:42:06.584367 32413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.578299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46775 (local address 127.23.213.195:40403)
0926 05:42:06.578419 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.578429 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.578533 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.579068 (+ 535us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.579069 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.579229 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:06.579275 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.579276 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.579857 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.579858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.580353 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.580354 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.580728 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.580729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.580786 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.581169 (+ 383us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:06.581175 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:06.581549 (+ 374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:06.583299 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.583300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.583300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.583472 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.583708 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:06.583709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:06.583709 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:06.583772 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:06.584030 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:06.584031 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:06.584076 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.584158 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.584267 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":66,"threads_started":1}
I20260926 05:42:06.584367 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.578360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:40403 (local address 127.23.213.254:46775)
0926 05:42:06.578400 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.578402 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.578506 (+ 104us) server_negotiation.cc:407] Connection header received
0926 05:42:06.578592 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.578593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.578601 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.578639 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.579340 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.579776 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.580459 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.580644 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.580702 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.581669 (+ 967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.581670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.581670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.581678 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.583195 (+ 1517us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.583559 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.583560 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.583561 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.583630 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.583845 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.583846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.583846 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.583957 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.584026 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.584197 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.584267 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:42:06.603536 31977 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:06.598300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51124 (local address 127.23.213.254:46775)
0926 05:42:06.598408 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.598410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.598419 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:06.598443 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.598443 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.598453 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.598505 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.599030 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.599513 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.600220 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.600425 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.600481 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.603439 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51124: BlockingRecv error: recv got EOF from 127.0.0.1:51124 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Sep 26 05:42:06 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401326, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:06.610733 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.604368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51140 (local address 127.23.213.254:46775)
0926 05:42:06.604425 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.604427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.604493 (+ 66us) server_negotiation.cc:407] Connection header received
0926 05:42:06.604527 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.604528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.604535 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.604594 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:06.605048 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.605482 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.606267 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.606436 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.606487 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.608455 (+ 1968us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:06.608456 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:06.608456 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:06.608462 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:06.609697 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.610057 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.610058 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.610058 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.610116 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:06.610273 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:06.610273 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:06.610274 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:06.610358 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:06.610422 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.610612 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.610675 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:42:06.612346 31923 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:42:06.613870 31923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:06.620651 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.616223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49600 (local address 127.23.213.194:45781)
0926 05:42:06.616642 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.616647 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.616656 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:06.616697 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.616699 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.616710 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.616820 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:06.617061 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.618231 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.619397 (+ 1166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.620133 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.620236 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.620297 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.620503 (+ 206us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.620551 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":83,"threads_started":1}
I20260926 05:42:06.620647 32430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.616110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45781 (local address 127.0.0.1:49600)
0926 05:42:06.616404 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.616447 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.616512 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.616828 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.616831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.616833 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:06.616899 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.616901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.618326 (+ 1425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.618327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.619302 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.619305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.620243 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.620244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.620341 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.620354 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.620478 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.620547 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"thread_start_us":58,"threads_started":1}
I20260926 05:42:06.621073 32429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.616158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38611 (local address 127.0.0.1:43298)
0926 05:42:06.616427 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.616453 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.616510 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.617261 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.617262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.617263 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:06.617296 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.617298 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.618489 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.618490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.619535 (+ 1045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.619537 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.620419 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.620420 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.620488 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.620491 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.620557 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.620591 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":255,"mutex_wait_us":1,"thread_start_us":68,"threads_started":1}
I20260926 05:42:06.621096 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.616266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43298 (local address 127.23.213.195:38611)
0926 05:42:06.617105 (+ 839us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.617110 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.617120 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:06.617159 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.617161 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.617175 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.617257 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:06.617355 (+ 98us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.618413 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.619624 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.620319 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.620399 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.620433 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.620939 (+ 506us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.620990 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":796,"thread_start_us":98,"threads_started":1}
I20260926 05:42:06.621603 32433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.616299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:46813 (local address 127.0.0.1:42916)
0926 05:42:06.616909 (+ 610us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.616919 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.616972 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.617194 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.617195 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.617202 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:06.617268 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.617270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.618234 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.618235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.619523 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.619525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.621386 (+ 1861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.621388 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.621450 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.621455 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.621485 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.621528 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":80,"threads_started":1}
I20260926 05:42:06.621511 32201 tablet_service.cc:1467] Processing CreateTablet for tablet 2de3c0589e5b45afa156fe34a4048736 (DEFAULT_TABLE table=test-table [id=80976654bf554601ab5f326a31f8efae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:06.621973 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de3c0589e5b45afa156fe34a4048736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:06.621989 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.616430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42916 (local address 127.23.213.193:46813)
0926 05:42:06.616616 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.616621 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.616984 (+ 363us) server_negotiation.cc:407] Connection header received
0926 05:42:06.617030 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.617032 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.617049 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.617130 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:06.617353 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.618151 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.620491 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.621284 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.621732 (+ 448us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.621816 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.621843 (+ 27us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.621889 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":89,"threads_started":1}
I20260926 05:42:06.621984 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 2de3c0589e5b45afa156fe34a4048736 (DEFAULT_TABLE table=test-table [id=80976654bf554601ab5f326a31f8efae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:06.622390 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de3c0589e5b45afa156fe34a4048736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:06.622633 32061 tablet_service.cc:1467] Processing CreateTablet for tablet 2de3c0589e5b45afa156fe34a4048736 (DEFAULT_TABLE table=test-table [id=80976654bf554601ab5f326a31f8efae]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:06.622947 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2de3c0589e5b45afa156fe34a4048736. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:06.625341 32435 tablet_bootstrap.cc:492] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Bootstrap starting.
I20260926 05:42:06.626003 32435 tablet_bootstrap.cc:654] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:06.626227 32436 tablet_bootstrap.cc:492] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Bootstrap starting.
I20260926 05:42:06.626353 32435 log.cc:824] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:06.626853 32436 tablet_bootstrap.cc:654] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:06.627107 32435 tablet_bootstrap.cc:492] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: No bootstrap required, opened a new log
I20260926 05:42:06.627183 32435 ts_tablet_manager.cc:1397] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:06.627195 32436 log.cc:824] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:06.627218 32438 tablet_bootstrap.cc:492] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Bootstrap starting.
I20260926 05:42:06.627875 32438 tablet_bootstrap.cc:654] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:06.628018 32436 tablet_bootstrap.cc:492] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: No bootstrap required, opened a new log
I20260926 05:42:06.628088 32436 ts_tablet_manager.cc:1397] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:06.628243 32438 log.cc:824] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:06.629017 32438 tablet_bootstrap.cc:492] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: No bootstrap required, opened a new log
I20260926 05:42:06.629098 32438 ts_tablet_manager.cc:1397] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:06.629256 32435 raft_consensus.cc:348] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.629432 32435 raft_consensus.cc:374] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:06.629472 32435 raft_consensus.cc:729] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cddb156244a4ab6b6833fa673046b05, State: Initialized, Role: FOLLOWER
I20260926 05:42:06.629580 32435 consensus_queue.cc:260] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.629840 32435 ts_tablet_manager.cc:1428] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:42:06.629977 32407 heartbeater.cc:503] Master 127.23.213.254:46775 was elected leader, sending a full tablet report...
I20260926 05:42:06.630246 32436 raft_consensus.cc:348] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.630396 32436 raft_consensus.cc:374] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:06.630440 32436 raft_consensus.cc:729] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea52a05957b41598459e22945bfab43, State: Initialized, Role: FOLLOWER
I20260926 05:42:06.630556 32436 consensus_queue.cc:260] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.630807 32436 ts_tablet_manager.cc:1428] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:42:06.630852 32267 heartbeater.cc:503] Master 127.23.213.254:46775 was elected leader, sending a full tablet report...
I20260926 05:42:06.631784 32438 raft_consensus.cc:348] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.631968 32438 raft_consensus.cc:374] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:06.631996 32438 raft_consensus.cc:729] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4128951d0e8e4051ac1597eda78a3747, State: Initialized, Role: FOLLOWER
I20260926 05:42:06.632138 32438 consensus_queue.cc:260] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.632393 32438 ts_tablet_manager.cc:1428] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:42:06.632452 32127 heartbeater.cc:503] Master 127.23.213.254:46775 was elected leader, sending a full tablet report...
W20260926 05:42:06.644331 32268 tablet.cc:2369] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:42:06.730136 32442 raft_consensus.cc:482] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:06.730289 32442 raft_consensus.cc:504] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.730674 32442 leader_election.cc:288] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611)
I20260926 05:42:06.734748 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.730951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:39979 (local address 127.23.213.193:46813)
0926 05:42:06.731000 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.731002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.731137 (+ 135us) server_negotiation.cc:407] Connection header received
0926 05:42:06.731177 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.731178 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.731187 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.731246 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:06.731383 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.732115 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.733656 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.734373 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.734454 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.734464 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.734625 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.734670 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:42:06.734748 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.730979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:35377 (local address 127.23.213.195:38611)
0926 05:42:06.731019 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.731021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.731029 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:06.731074 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.731075 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.731082 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.731164 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:06.731336 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.732110 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.733654 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.734323 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.734444 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.734456 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.734624 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.734670 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:42:06.734756 32444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.730853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:46813 (local address 127.23.213.194:39979)
0926 05:42:06.731068 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.731075 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.731115 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.731255 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.731255 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.731256 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:06.731312 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.731314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.732196 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.732197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.733528 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.733531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.734451 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.734451 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.734527 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.734531 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.734597 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.734644 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":106,"thread_start_us":105,"threads_started":1}
I20260926 05:42:06.734763 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.730853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38611 (local address 127.23.213.194:35377)
0926 05:42:06.730891 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:06.730903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:06.730953 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:06.731180 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:06.731181 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:06.731183 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:06.731256 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.731257 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.732196 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.732197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.733528 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:06.733531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.734444 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.734445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:06.734530 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.734531 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:42:06.734592 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:42:06.734644 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18,"mutex_wait_us":96}
I20260926 05:42:06.735132 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de3c0589e5b45afa156fe34a4048736" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cddb156244a4ab6b6833fa673046b05" is_pre_election: true
I20260926 05:42:06.735157 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de3c0589e5b45afa156fe34a4048736" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747" is_pre_election: true
I20260926 05:42:06.735288 32081 raft_consensus.cc:2454] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 0.
I20260926 05:42:06.735288 32361 raft_consensus.cc:2454] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 0.
I20260926 05:42:06.735529 32154 leader_election.cc:302] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4128951d0e8e4051ac1597eda78a3747, fea52a05957b41598459e22945bfab43; no voters:
I20260926 05:42:06.735738 32442 raft_consensus.cc:2781] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:06.735816 32442 raft_consensus.cc:482] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:06.735843 32442 raft_consensus.cc:3037] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:06.737088 32442 raft_consensus.cc:504] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.737267 32442 leader_election.cc:288] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 election: Requested vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611)
I20260926 05:42:06.737434 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de3c0589e5b45afa156fe34a4048736" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cddb156244a4ab6b6833fa673046b05"
I20260926 05:42:06.737447 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2de3c0589e5b45afa156fe34a4048736" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747"
I20260926 05:42:06.737542 32081 raft_consensus.cc:3037] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:06.737533 32361 raft_consensus.cc:3037] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:06.738381 32361 raft_consensus.cc:2454] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 1.
I20260926 05:42:06.738413 32081 raft_consensus.cc:2454] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 1.
I20260926 05:42:06.738600 32154 leader_election.cc:302] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4128951d0e8e4051ac1597eda78a3747, fea52a05957b41598459e22945bfab43; no voters:
I20260926 05:42:06.738713 32442 raft_consensus.cc:2781] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:06.738914 32442 raft_consensus.cc:686] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 1 LEADER]: Becoming Leader. State: Replica: fea52a05957b41598459e22945bfab43, State: Running, Role: LEADER
I20260926 05:42:06.739042 32442 consensus_queue.cc:237] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.739995 31924 catalog_manager.cc:5495] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 reported cstate change: term changed from 0 to 1, leader changed from <none> to fea52a05957b41598459e22945bfab43 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "fea52a05957b41598459e22945bfab43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } health_report { overall_health: UNKNOWN } } }
W20260926 05:42:06.743665 32128 tablet.cc:2369] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:42:06.770105 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:06.767516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49614 (local address 127.23.213.194:45781)
0926 05:42:06.767591 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:06.767593 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:06.767603 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:06.767683 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:06.767684 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:06.767705 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:06.767795 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:06.767941 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.768475 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.769241 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:06.769451 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:06.769535 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:06.769711 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:06.769885 (+ 174us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:06.769979 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:06.769999 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 05:42:06.770019 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 05:42:06.772001 32361 raft_consensus.cc:1261] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Refusing update from remote peer fea52a05957b41598459e22945bfab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:06.772027 32081 raft_consensus.cc:1261] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Refusing update from remote peer fea52a05957b41598459e22945bfab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:06.772356 32445 consensus_queue.cc:1035] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:06.772519 32442 consensus_queue.cc:1035] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:06.774943 32450 mvcc.cc:205] Tried to move back new op lower bound from 7333483834455531520 to 7333483834323877888. Current Snapshot: MvccSnapshot[applied={T|T < 7333483834455531520}]
I20260926 05:42:06.777073 31924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:06.777179 31924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:06.777911 31924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Sending DeleteTablet for 3 replicas of tablet 2de3c0589e5b45afa156fe34a4048736
I20260926 05:42:06.778249 32061 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de3c0589e5b45afa156fe34a4048736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:06 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42916
I20260926 05:42:06.778249 32341 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de3c0589e5b45afa156fe34a4048736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:06 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43298
I20260926 05:42:06.778473 32201 tablet_service.cc:1514] Processing DeleteTablet for tablet 2de3c0589e5b45afa156fe34a4048736 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:06 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:49600
I20260926 05:42:06.778623 32459 tablet_replica.cc:330] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: stopping tablet replica
I20260926 05:42:06.778676 32458 tablet_replica.cc:330] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: stopping tablet replica
I20260926 05:42:06.778720 32459 raft_consensus.cc:2229] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:06.778764 32458 raft_consensus.cc:2229] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:06.778728 31924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51140:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 05:42:06.778854 32458 raft_consensus.cc:2258] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 05:42:06.778877 31924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:06.779235 32459 raft_consensus.cc:2258] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:06.779548 32460 tablet_replica.cc:330] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: stopping tablet replica
I20260926 05:42:06.779680 32460 raft_consensus.cc:2229] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:42:06.779680 32459 ts_tablet_manager.cc:1905] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:06.779840 32460 raft_consensus.cc:2258] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:06.780417 32460 ts_tablet_manager.cc:1905] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:06.780437 32061 tablet_service.cc:1467] Processing CreateTablet for tablet 3a601c1410314b108d1f04be1142129e (DEFAULT_TABLE table=test-table [id=447b312b3aba4368b8824aed3aa322d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:06.780562 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a601c1410314b108d1f04be1142129e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:06.781356 32201 tablet_service.cc:1467] Processing CreateTablet for tablet 3a601c1410314b108d1f04be1142129e (DEFAULT_TABLE table=test-table [id=447b312b3aba4368b8824aed3aa322d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:06.781510 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 3a601c1410314b108d1f04be1142129e (DEFAULT_TABLE table=test-table [id=447b312b3aba4368b8824aed3aa322d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:06.781598 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a601c1410314b108d1f04be1142129e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:06.781622 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3a601c1410314b108d1f04be1142129e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:06.782531 32438 tablet_bootstrap.cc:492] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: Bootstrap starting.
I20260926 05:42:06.782897 32438 tablet_bootstrap.cc:654] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:06.782985 32436 tablet_bootstrap.cc:492] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: Bootstrap starting.
I20260926 05:42:06.783403 32436 tablet_bootstrap.cc:654] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:06.783627 32458 ts_tablet_manager.cc:1905] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:06.783730 32435 tablet_bootstrap.cc:492] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Bootstrap starting.
I20260926 05:42:06.784060 32460 ts_tablet_manager.cc:1918] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:06.784142 32435 tablet_bootstrap.cc:654] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:06.784227 32460 log.cc:1196] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/2de3c0589e5b45afa156fe34a4048736
I20260926 05:42:06.781502 32459 ts_tablet_manager.cc:1918] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:06.784610 32459 log.cc:1196] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/2de3c0589e5b45afa156fe34a4048736
I20260926 05:42:06.784687 32460 ts_tablet_manager.cc:1939] T 2de3c0589e5b45afa156fe34a4048736 P fea52a05957b41598459e22945bfab43: Deleting consensus metadata
I20260926 05:42:06.784865 32459 ts_tablet_manager.cc:1939] T 2de3c0589e5b45afa156fe34a4048736 P 4cddb156244a4ab6b6833fa673046b05: Deleting consensus metadata
I20260926 05:42:06.785027 32458 ts_tablet_manager.cc:1918] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:06.785185 31910 catalog_manager.cc:4841] TS fea52a05957b41598459e22945bfab43 (127.23.213.194:45781): tablet 2de3c0589e5b45afa156fe34a4048736 (table test-table [id=80976654bf554601ab5f326a31f8efae]) successfully deleted
I20260926 05:42:06.785218 31909 catalog_manager.cc:4841] TS 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611): tablet 2de3c0589e5b45afa156fe34a4048736 (table test-table [id=80976654bf554601ab5f326a31f8efae]) successfully deleted
I20260926 05:42:06.785396 32458 log.cc:1196] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/2de3c0589e5b45afa156fe34a4048736
I20260926 05:42:06.785655 32436 tablet_bootstrap.cc:492] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: No bootstrap required, opened a new log
I20260926 05:42:06.785686 32435 tablet_bootstrap.cc:492] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: No bootstrap required, opened a new log
I20260926 05:42:06.785696 32458 ts_tablet_manager.cc:1939] T 2de3c0589e5b45afa156fe34a4048736 P 4128951d0e8e4051ac1597eda78a3747: Deleting consensus metadata
I20260926 05:42:06.785699 32436 ts_tablet_manager.cc:1397] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 05:42:06.785722 32435 ts_tablet_manager.cc:1397] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:06.785877 32436 raft_consensus.cc:348] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.785961 32436 raft_consensus.cc:374] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:06.785959 32435 raft_consensus.cc:348] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.786026 32435 raft_consensus.cc:374] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:06.786049 32435 raft_consensus.cc:729] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cddb156244a4ab6b6833fa673046b05, State: Initialized, Role: FOLLOWER
I20260926 05:42:06.786072 31910 catalog_manager.cc:4841] TS 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813): tablet 2de3c0589e5b45afa156fe34a4048736 (table test-table [id=80976654bf554601ab5f326a31f8efae]) successfully deleted
I20260926 05:42:06.786091 32435 consensus_queue.cc:260] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.785983 32436 raft_consensus.cc:729] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea52a05957b41598459e22945bfab43, State: Initialized, Role: FOLLOWER
I20260926 05:42:06.786178 32435 ts_tablet_manager.cc:1428] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:06.786161 32436 consensus_queue.cc:260] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.786273 32436 ts_tablet_manager.cc:1428] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:06.787006 32438 tablet_bootstrap.cc:492] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: No bootstrap required, opened a new log
I20260926 05:42:06.787060 32438 ts_tablet_manager.cc:1397] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 05:42:06.787199 32438 raft_consensus.cc:348] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.787284 32438 raft_consensus.cc:374] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:06.787318 32438 raft_consensus.cc:729] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4128951d0e8e4051ac1597eda78a3747, State: Initialized, Role: FOLLOWER
I20260926 05:42:06.787367 32438 consensus_queue.cc:260] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.787477 32438 ts_tablet_manager.cc:1428] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
W20260926 05:42:06.824808 32408 tablet.cc:2369] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:42:06.986179 32453 raft_consensus.cc:482] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:06.986280 32453 raft_consensus.cc:504] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.986474 32453 leader_election.cc:288] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611)
I20260926 05:42:06.986754 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a601c1410314b108d1f04be1142129e" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747" is_pre_election: true
I20260926 05:42:06.986754 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a601c1410314b108d1f04be1142129e" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cddb156244a4ab6b6833fa673046b05" is_pre_election: true
I20260926 05:42:06.986872 32081 raft_consensus.cc:2454] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 0.
I20260926 05:42:06.986907 32361 raft_consensus.cc:2454] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 0.
I20260926 05:42:06.987053 32154 leader_election.cc:302] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4128951d0e8e4051ac1597eda78a3747, fea52a05957b41598459e22945bfab43; no voters:
I20260926 05:42:06.987179 32453 raft_consensus.cc:2781] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:06.987236 32453 raft_consensus.cc:482] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:06.987258 32453 raft_consensus.cc:3037] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:06.988011 32453 raft_consensus.cc:504] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.988153 32453 leader_election.cc:288] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 election: Requested vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611)
I20260926 05:42:06.988345 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a601c1410314b108d1f04be1142129e" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cddb156244a4ab6b6833fa673046b05"
I20260926 05:42:06.988433 32361 raft_consensus.cc:3037] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:06.988682 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3a601c1410314b108d1f04be1142129e" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747"
I20260926 05:42:06.988756 32081 raft_consensus.cc:3037] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:06.989238 32361 raft_consensus.cc:2454] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 1.
I20260926 05:42:06.989395 32153 leader_election.cc:302] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cddb156244a4ab6b6833fa673046b05, fea52a05957b41598459e22945bfab43; no voters:
I20260926 05:42:06.989477 32453 raft_consensus.cc:2781] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:06.989511 32453 raft_consensus.cc:686] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 1 LEADER]: Becoming Leader. State: Replica: fea52a05957b41598459e22945bfab43, State: Running, Role: LEADER
I20260926 05:42:06.989537 32081 raft_consensus.cc:2454] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 1.
I20260926 05:42:06.989562 32453 consensus_queue.cc:237] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:06.990222 31923 catalog_manager.cc:5495] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 reported cstate change: term changed from 0 to 1, leader changed from <none> to fea52a05957b41598459e22945bfab43 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "fea52a05957b41598459e22945bfab43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } health_report { overall_health: UNKNOWN } } }
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401325, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.254@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:07.033713 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.026479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46775 (local address 127.0.0.1:51146)
0926 05:42:07.026689 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:07.026700 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:07.026768 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:07.026889 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:07.026890 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:07.027200 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:07.027247 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.027249 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.027949 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.027951 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.028640 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.028642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.029002 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.029003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.029049 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.029802 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:07.029808 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:07.031222 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:07.032623 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:07.032629 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:07.032638 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:07.032878 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:07.033054 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:07.033055 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:07.033055 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:07.033113 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:07.033318 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:07.033325 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:07.033371 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 05:42:07.033428 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 05:42:07.033523 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"thread_start_us":104,"threads_started":1}
I20260926 05:42:07.033773 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.026538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51146 (local address 127.23.213.254:46775)
0926 05:42:07.026643 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.026645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.026771 (+ 126us) server_negotiation.cc:407] Connection header received
0926 05:42:07.026810 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.026810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.026819 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.026869 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:07.027316 (+ 447us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.027860 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.028730 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.028922 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.028973 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.031296 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:07.031297 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:07.031297 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:07.031303 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:07.032547 (+ 1244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:07.032945 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:07.032946 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:07.032946 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:07.032993 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:07.033166 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:07.033167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:07.033167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:07.033260 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:07.033308 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.033459 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.033570 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":77}
I20260926 05:42:07.036217 31923 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:51146:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 05:42:07.036440 31923 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:07.037899 32061 tablet_service.cc:1467] Processing CreateTablet for tablet 76842d46c5c644a292c1971ab6183daf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5262ce26948e427ab9f7e8730b0ce2ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:42:07.037906 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 76842d46c5c644a292c1971ab6183daf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5262ce26948e427ab9f7e8730b0ce2ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:42:07.037891 32201 tablet_service.cc:1467] Processing CreateTablet for tablet 76842d46c5c644a292c1971ab6183daf (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5262ce26948e427ab9f7e8730b0ce2ac]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 05:42:07.038038 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76842d46c5c644a292c1971ab6183daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.038051 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76842d46c5c644a292c1971ab6183daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.038059 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 76842d46c5c644a292c1971ab6183daf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.039142 32438 tablet_bootstrap.cc:492] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: Bootstrap starting.
I20260926 05:42:07.039465 32436 tablet_bootstrap.cc:492] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: Bootstrap starting.
I20260926 05:42:07.039557 32438 tablet_bootstrap.cc:654] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.039601 32435 tablet_bootstrap.cc:492] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: Bootstrap starting.
I20260926 05:42:07.039796 32436 tablet_bootstrap.cc:654] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.039980 32435 tablet_bootstrap.cc:654] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.040207 32438 tablet_bootstrap.cc:492] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: No bootstrap required, opened a new log
I20260926 05:42:07.040249 32438 ts_tablet_manager.cc:1397] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:07.040442 32438 raft_consensus.cc:348] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.040525 32438 raft_consensus.cc:374] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.040556 32438 raft_consensus.cc:729] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4128951d0e8e4051ac1597eda78a3747, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.040601 32438 consensus_queue.cc:260] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.040619 32436 tablet_bootstrap.cc:492] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: No bootstrap required, opened a new log
I20260926 05:42:07.040634 32435 tablet_bootstrap.cc:492] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: No bootstrap required, opened a new log
I20260926 05:42:07.040666 32436 ts_tablet_manager.cc:1397] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:07.040670 32435 ts_tablet_manager.cc:1397] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:07.040691 32438 ts_tablet_manager.cc:1428] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.040805 32436 raft_consensus.cc:348] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.040889 32436 raft_consensus.cc:374] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.040858 32435 raft_consensus.cc:348] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.040911 32436 raft_consensus.cc:729] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea52a05957b41598459e22945bfab43, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.040920 32435 raft_consensus.cc:374] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.040941 32435 raft_consensus.cc:729] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cddb156244a4ab6b6833fa673046b05, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.040956 32436 consensus_queue.cc:260] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.040980 32435 consensus_queue.cc:260] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.040988 32443 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.041078 32436 ts_tablet_manager.cc:1428] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:07.041082 32443 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:42:07.041092 32435 ts_tablet_manager.cc:1428] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.041203 32453 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.041301 32453 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:42:07.041332 32441 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.041427 32441 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:42:07.158625 32441 raft_consensus.cc:482] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:07.158722 32441 raft_consensus.cc:504] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.158999 32441 leader_election.cc:288] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), fea52a05957b41598459e22945bfab43 (127.23.213.194:45781)
I20260926 05:42:07.163094 32434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.159268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:59035 (local address 127.23.213.193:46813)
0926 05:42:07.159342 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.159344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.159467 (+ 123us) server_negotiation.cc:407] Connection header received
0926 05:42:07.159499 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.159500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.159511 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.159585 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:07.159800 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.160585 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.162162 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.162708 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.162786 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.162794 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.162990 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.163027 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:42:07.163096 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.159474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:43595 (local address 127.23.213.194:45781)
0926 05:42:07.159534 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.159536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.159555 (+ 19us) server_negotiation.cc:407] Connection header received
0926 05:42:07.159669 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.159670 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.159687 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.159729 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:07.159883 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.160580 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.162162 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.162708 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.162786 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.162794 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.162980 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.163023 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:42:07.163111 32474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.159192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:46813 (local address 127.23.213.195:59035)
0926 05:42:07.159353 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:07.159365 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:07.159439 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:07.159605 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:07.159606 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:07.159609 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:07.159699 (+ 90us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.159701 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.160672 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.160673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.162069 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.162072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.162799 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.162800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.162915 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.162924 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 05:42:07.162974 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:42:07.163044 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":13,"thread_start_us":80,"threads_started":1}
I20260926 05:42:07.163111 32475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.159383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45781 (local address 127.23.213.195:43595)
0926 05:42:07.159505 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:07.159515 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:07.159589 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:07.159757 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:07.159758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:07.159760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:07.159814 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.159816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.160672 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.160673 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.162069 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.162072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.162798 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.162799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.162880 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.162885 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:42:07.162946 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 05:42:07.162998 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"mutex_wait_us":71,"thread_start_us":55,"threads_started":1}
I20260926 05:42:07.163343 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76842d46c5c644a292c1971ab6183daf" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747" is_pre_election: true
I20260926 05:42:07.163422 32081 raft_consensus.cc:2454] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 0.
I20260926 05:42:07.163599 32294 leader_election.cc:302] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4128951d0e8e4051ac1597eda78a3747, 4cddb156244a4ab6b6833fa673046b05; no voters:
I20260926 05:42:07.163652 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76842d46c5c644a292c1971ab6183daf" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea52a05957b41598459e22945bfab43" is_pre_election: true
I20260926 05:42:07.163757 32441 raft_consensus.cc:2781] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:07.163800 32221 raft_consensus.cc:2454] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 0.
I20260926 05:42:07.163810 32441 raft_consensus.cc:482] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:07.163842 32441 raft_consensus.cc:3037] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.164549 32441 raft_consensus.cc:504] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.164691 32441 leader_election.cc:288] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 election: Requested vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), fea52a05957b41598459e22945bfab43 (127.23.213.194:45781)
I20260926 05:42:07.164960 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76842d46c5c644a292c1971ab6183daf" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea52a05957b41598459e22945bfab43"
I20260926 05:42:07.164961 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "76842d46c5c644a292c1971ab6183daf" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747"
I20260926 05:42:07.165045 32221 raft_consensus.cc:3037] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.165046 32081 raft_consensus.cc:3037] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.165825 32081 raft_consensus.cc:2454] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 1.
I20260926 05:42:07.165830 32221 raft_consensus.cc:2454] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 1.
I20260926 05:42:07.166009 32294 leader_election.cc:302] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4128951d0e8e4051ac1597eda78a3747, 4cddb156244a4ab6b6833fa673046b05; no voters:
I20260926 05:42:07.166133 32441 raft_consensus.cc:2781] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:07.166282 32441 raft_consensus.cc:686] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [term 1 LEADER]: Becoming Leader. State: Replica: 4cddb156244a4ab6b6833fa673046b05, State: Running, Role: LEADER
I20260926 05:42:07.166366 32441 consensus_queue.cc:237] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.166705 32476 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: TxnStatusTablet state changed. Reason: New leader 4cddb156244a4ab6b6833fa673046b05. Latest consensus state: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.166821 32476 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:42:07.167002 32478 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 05:42:07.167104 32478 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 05:42:07.167136 31922 catalog_manager.cc:5495] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195). New cstate: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } health_report { overall_health: HEALTHY } } }
I20260926 05:42:07.190374 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.187897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.23.213.195:38611)
0926 05:42:07.187948 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.187950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.187960 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:07.188044 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.188045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.188065 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.188129 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:07.188265 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.188757 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.189517 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.189731 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.189840 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.190038 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.190148 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.190229 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.190272 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.190293 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:42:07.190389 32470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.187828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38611 (local address 127.0.0.1:43304)
0926 05:42:07.187901 (+ 73us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:07.187913 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:07.187965 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:07.188135 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:07.188135 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:07.188137 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:42:07.188190 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.188192 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.188848 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.188850 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.189429 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.189430 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.189841 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.189841 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.189910 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.189933 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.190214 (+ 281us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.190224 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 05:42:07.190269 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:42:07.190320 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:42:07.191612 32081 raft_consensus.cc:1261] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Refusing update from remote peer 4cddb156244a4ab6b6833fa673046b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.191671 32221 raft_consensus.cc:1261] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Refusing update from remote peer 4cddb156244a4ab6b6833fa673046b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.191915 32441 consensus_queue.cc:1035] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.192055 32476 consensus_queue.cc:1035] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.192613 32453 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: TxnStatusTablet state changed. Reason: New leader 4cddb156244a4ab6b6833fa673046b05. Latest consensus state: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.192701 32453 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:42:07.192654 32443 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: TxnStatusTablet state changed. Reason: New leader 4cddb156244a4ab6b6833fa673046b05. Latest consensus state: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.192739 32443 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:42:07.193503 32476 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.193594 32476 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:42:07.193856 32476 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.193930 32476 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P 4cddb156244a4ab6b6833fa673046b05: This TxnStatusTablet replica's current role is: LEADER
I20260926 05:42:07.194291 32453 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.194372 32453 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P fea52a05957b41598459e22945bfab43: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:42:07.194998 32449 mvcc.cc:205] Tried to move back new op lower bound from 7333483836174385152 to 7333483836073824256. Current Snapshot: MvccSnapshot[applied={T|T < 7333483836174385152}]
I20260926 05:42:07.195072 32443 tablet_replica.cc:439] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } } }
I20260926 05:42:07.195158 32443 tablet_replica.cc:442] T 76842d46c5c644a292c1971ab6183daf P 4128951d0e8e4051ac1597eda78a3747: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 05:42:07.199790 32432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.197156 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:40803 (local address 127.23.213.195:38611)
0926 05:42:07.197198 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.197200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.197290 (+ 90us) server_negotiation.cc:407] Connection header received
0926 05:42:07.197363 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.197364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.197380 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.197435 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:07.197556 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.198068 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.199203 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.199375 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.199426 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.199611 (+ 185us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.199635 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.199672 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.199725 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.199741 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:42:07.199856 32486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.197071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38611 (local address 127.23.213.194:40803)
0926 05:42:07.197228 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:07.197240 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:07.197302 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:07.197436 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:07.197437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:07.197439 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:42:07.197493 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.197495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.198141 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.198142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.199123 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.199126 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.199455 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.199456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.199516 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.199532 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.199673 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.199675 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:42:07.199723 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:42:07.199770 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":88,"threads_started":1}
I20260926 05:42:07.202239 32081 raft_consensus.cc:1261] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Refusing update from remote peer fea52a05957b41598459e22945bfab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.202246 32361 raft_consensus.cc:1261] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Refusing update from remote peer fea52a05957b41598459e22945bfab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.202445 32453 consensus_queue.cc:1035] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.202600 32445 consensus_queue.cc:1035] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.204109 32448 mvcc.cc:205] Tried to move back new op lower bound from 7333483836218949632 to 7333483835349544960. Current Snapshot: MvccSnapshot[applied={T|T < 7333483836218949632}]
I20260926 05:42:07.211476 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.208623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:51869 (local address 127.23.213.194:45781)
0926 05:42:07.208661 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.208664 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.208772 (+ 108us) server_negotiation.cc:407] Connection header received
0926 05:42:07.208838 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.208839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.208848 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.208887 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:07.209029 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.209561 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.210658 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.210998 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.211070 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.211251 (+ 181us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.211288 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.211351 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.211386 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.211408 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:42:07.211484 32490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.208561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:45781 (local address 127.23.213.195:51869)
0926 05:42:07.208716 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:07.208727 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:07.208782 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:07.208898 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:07.208899 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:07.208900 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:42:07.208957 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.208959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.209637 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.209638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.210581 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.210583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.211101 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.211102 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.211157 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.211169 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.211348 (+ 179us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.211350 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:42:07.211380 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 05:42:07.211416 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260926 05:42:07.220096 31922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:07.220211 31922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51140:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:07.220847 31922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Sending DeleteTablet for 3 replicas of tablet 3a601c1410314b108d1f04be1142129e
I20260926 05:42:07.221071 32061 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a601c1410314b108d1f04be1142129e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42916
I20260926 05:42:07.221163 32201 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a601c1410314b108d1f04be1142129e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:49600
I20260926 05:42:07.221249 32341 tablet_service.cc:1514] Processing DeleteTablet for tablet 3a601c1410314b108d1f04be1142129e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43298
I20260926 05:42:07.221275 32458 tablet_replica.cc:330] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: stopping tablet replica
I20260926 05:42:07.221271 32460 tablet_replica.cc:330] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: stopping tablet replica
I20260926 05:42:07.221329 32459 tablet_replica.cc:330] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: stopping tablet replica
I20260926 05:42:07.221333 32458 raft_consensus.cc:2229] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:07.221336 32460 raft_consensus.cc:2229] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:42:07.221369 32458 raft_consensus.cc:2258] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.221379 32459 raft_consensus.cc:2229] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:07.221390 32460 raft_consensus.cc:2258] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.221416 32459 raft_consensus.cc:2258] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.221701 32460 ts_tablet_manager.cc:1905] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.221720 32459 ts_tablet_manager.cc:1905] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.221720 32458 ts_tablet_manager.cc:1905] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.223299 32460 ts_tablet_manager.cc:1918] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:42:07.223361 32460 log.cc:1196] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/3a601c1410314b108d1f04be1142129e
I20260926 05:42:07.223599 32460 ts_tablet_manager.cc:1939] T 3a601c1410314b108d1f04be1142129e P fea52a05957b41598459e22945bfab43: Deleting consensus metadata
I20260926 05:42:07.225247 31910 catalog_manager.cc:4841] TS fea52a05957b41598459e22945bfab43 (127.23.213.194:45781): tablet 3a601c1410314b108d1f04be1142129e (table test-table [id=447b312b3aba4368b8824aed3aa322d3]) successfully deleted
I20260926 05:42:07.225876 32459 ts_tablet_manager.cc:1918] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:42:07.225937 32459 log.cc:1196] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/3a601c1410314b108d1f04be1142129e
I20260926 05:42:07.226151 32459 ts_tablet_manager.cc:1939] T 3a601c1410314b108d1f04be1142129e P 4cddb156244a4ab6b6833fa673046b05: Deleting consensus metadata
I20260926 05:42:07.226465 32458 ts_tablet_manager.cc:1918] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:42:07.226467 31909 catalog_manager.cc:4841] TS 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611): tablet 3a601c1410314b108d1f04be1142129e (table test-table [id=447b312b3aba4368b8824aed3aa322d3]) successfully deleted
I20260926 05:42:07.226521 32458 log.cc:1196] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/3a601c1410314b108d1f04be1142129e
I20260926 05:42:07.226722 32458 ts_tablet_manager.cc:1939] T 3a601c1410314b108d1f04be1142129e P 4128951d0e8e4051ac1597eda78a3747: Deleting consensus metadata
I20260926 05:42:07.227028 31910 catalog_manager.cc:4841] TS 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813): tablet 3a601c1410314b108d1f04be1142129e (table test-table [id=447b312b3aba4368b8824aed3aa322d3]) successfully deleted
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[31879](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:42:07.246114 31977 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:07.239707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51156 (local address 127.23.213.254:46775)
0926 05:42:07.239763 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.239765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.239875 (+ 110us) server_negotiation.cc:407] Connection header received
0926 05:42:07.239934 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.239935 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.239947 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.240012 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:07.240558 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.241325 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.242185 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.242487 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.242570 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.246046 (+ 3476us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51156: BlockingRecv error: recv got EOF from 127.0.0.1:51156 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[31879](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401327, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM
I20260926 05:42:07.254485 31977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.246964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51164 (local address 127.23.213.254:46775)
0926 05:42:07.247002 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.247004 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.247017 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:42:07.247095 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.247095 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.247104 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.247148 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:07.247716 (+ 568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.248182 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.249179 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.249351 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.249396 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.251862 (+ 2466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:07.251863 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:07.251863 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:07.251871 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:07.253387 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:07.253709 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:07.253711 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:07.253711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:07.253794 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:07.253987 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:07.253988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:07.253988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:07.254123 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:07.254218 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.254306 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.254393 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:42:07.255986 31922 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:42:07.256135 31922 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:07.257529 32061 tablet_service.cc:1467] Processing CreateTablet for tablet e910f11234fb44bebb85fe92764e5f8a (DEFAULT_TABLE table=test-table [id=b3e2d8137bea48c2be4884411bf47ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:07.257509 32201 tablet_service.cc:1467] Processing CreateTablet for tablet e910f11234fb44bebb85fe92764e5f8a (DEFAULT_TABLE table=test-table [id=b3e2d8137bea48c2be4884411bf47ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:07.257597 32341 tablet_service.cc:1467] Processing CreateTablet for tablet e910f11234fb44bebb85fe92764e5f8a (DEFAULT_TABLE table=test-table [id=b3e2d8137bea48c2be4884411bf47ec3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:07.257655 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e910f11234fb44bebb85fe92764e5f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.257655 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e910f11234fb44bebb85fe92764e5f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.257723 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e910f11234fb44bebb85fe92764e5f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.258976 32438 tablet_bootstrap.cc:492] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: Bootstrap starting.
I20260926 05:42:07.259013 32436 tablet_bootstrap.cc:492] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: Bootstrap starting.
I20260926 05:42:07.259169 32435 tablet_bootstrap.cc:492] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: Bootstrap starting.
I20260926 05:42:07.259325 32436 tablet_bootstrap.cc:654] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.259325 32438 tablet_bootstrap.cc:654] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.259591 32435 tablet_bootstrap.cc:654] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.260108 32436 tablet_bootstrap.cc:492] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: No bootstrap required, opened a new log
I20260926 05:42:07.260160 32436 ts_tablet_manager.cc:1397] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:07.260290 32435 tablet_bootstrap.cc:492] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: No bootstrap required, opened a new log
I20260926 05:42:07.260334 32435 ts_tablet_manager.cc:1397] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:07.260313 32436 raft_consensus.cc:348] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.260383 32436 raft_consensus.cc:374] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.260412 32436 raft_consensus.cc:729] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea52a05957b41598459e22945bfab43, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.260456 32436 consensus_queue.cc:260] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.260475 32435 raft_consensus.cc:348] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.260536 32435 raft_consensus.cc:374] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.260556 32435 raft_consensus.cc:729] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cddb156244a4ab6b6833fa673046b05, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.260564 32436 ts_tablet_manager.cc:1428] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.260597 32435 consensus_queue.cc:260] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.260686 32435 ts_tablet_manager.cc:1428] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.261152 32438 tablet_bootstrap.cc:492] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: No bootstrap required, opened a new log
I20260926 05:42:07.261205 32438 ts_tablet_manager.cc:1397] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:07.261334 32438 raft_consensus.cc:348] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.261417 32438 raft_consensus.cc:374] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.261439 32438 raft_consensus.cc:729] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4128951d0e8e4051ac1597eda78a3747, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.261487 32438 consensus_queue.cc:260] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.261579 32438 ts_tablet_manager.cc:1428] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.488579 32442 raft_consensus.cc:482] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:07.488672 32442 raft_consensus.cc:504] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.488832 32442 leader_election.cc:288] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611)
I20260926 05:42:07.489110 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e910f11234fb44bebb85fe92764e5f8a" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cddb156244a4ab6b6833fa673046b05" is_pre_election: true
I20260926 05:42:07.489131 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e910f11234fb44bebb85fe92764e5f8a" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747" is_pre_election: true
I20260926 05:42:07.489214 32081 raft_consensus.cc:2454] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 0.
I20260926 05:42:07.489213 32361 raft_consensus.cc:2454] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 0.
I20260926 05:42:07.489388 32153 leader_election.cc:302] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cddb156244a4ab6b6833fa673046b05, fea52a05957b41598459e22945bfab43; no voters:
I20260926 05:42:07.489497 32442 raft_consensus.cc:2781] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:07.489537 32442 raft_consensus.cc:482] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:07.489563 32442 raft_consensus.cc:3037] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.490304 32442 raft_consensus.cc:504] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.490434 32442 leader_election.cc:288] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 election: Requested vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611)
I20260926 05:42:07.490635 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e910f11234fb44bebb85fe92764e5f8a" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747"
I20260926 05:42:07.490635 32361 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e910f11234fb44bebb85fe92764e5f8a" candidate_uuid: "fea52a05957b41598459e22945bfab43" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4cddb156244a4ab6b6833fa673046b05"
I20260926 05:42:07.490736 32081 raft_consensus.cc:3037] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.490736 32361 raft_consensus.cc:3037] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.491482 32081 raft_consensus.cc:2454] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 1.
I20260926 05:42:07.491482 32361 raft_consensus.cc:2454] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fea52a05957b41598459e22945bfab43 in term 1.
I20260926 05:42:07.491698 32154 leader_election.cc:302] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 4128951d0e8e4051ac1597eda78a3747, 4cddb156244a4ab6b6833fa673046b05, fea52a05957b41598459e22945bfab43; no voters:
I20260926 05:42:07.491811 32442 raft_consensus.cc:2781] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:07.491881 32442 raft_consensus.cc:686] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 1 LEADER]: Becoming Leader. State: Replica: fea52a05957b41598459e22945bfab43, State: Running, Role: LEADER
I20260926 05:42:07.491938 32442 consensus_queue.cc:237] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.492480 31924 catalog_manager.cc:5495] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 reported cstate change: term changed from 0 to 1, leader changed from <none> to fea52a05957b41598459e22945bfab43 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "fea52a05957b41598459e22945bfab43" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } health_report { overall_health: UNKNOWN } } }
I20260926 05:42:07.545608 32431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.543175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44800 (local address 127.23.213.194:45781)
0926 05:42:07.543221 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.543223 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.543231 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:07.543299 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.543300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.543314 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.543383 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:07.543491 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.544006 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.544866 (+ 860us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.545089 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.545153 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.545331 (+ 178us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.545366 (+ 35us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.545435 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.545498 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.545521 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:42:07.546403 32361 raft_consensus.cc:1261] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Refusing update from remote peer fea52a05957b41598459e22945bfab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.546442 32081 raft_consensus.cc:1261] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Refusing update from remote peer fea52a05957b41598459e22945bfab43: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.546679 32445 consensus_queue.cc:1035] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.546826 32442 consensus_queue.cc:1035] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.549238 31924 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:07.549314 31924 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:07.549885 31924 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Sending DeleteTablet for 3 replicas of tablet e910f11234fb44bebb85fe92764e5f8a
I20260926 05:42:07.550134 32341 tablet_service.cc:1514] Processing DeleteTablet for tablet e910f11234fb44bebb85fe92764e5f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43298
I20260926 05:42:07.550155 32201 tablet_service.cc:1514] Processing DeleteTablet for tablet e910f11234fb44bebb85fe92764e5f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:49600
I20260926 05:42:07.550158 32061 tablet_service.cc:1514] Processing DeleteTablet for tablet e910f11234fb44bebb85fe92764e5f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42916
I20260926 05:42:07.550227 32459 tablet_replica.cc:330] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: stopping tablet replica
I20260926 05:42:07.550263 32460 tablet_replica.cc:330] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: stopping tablet replica
I20260926 05:42:07.550290 32459 raft_consensus.cc:2229] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:07.550313 32460 raft_consensus.cc:2229] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:42:07.550343 32459 raft_consensus.cc:2258] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.550376 32460 raft_consensus.cc:2258] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.550473 32458 tablet_replica.cc:330] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: stopping tablet replica
I20260926 05:42:07.550532 32458 raft_consensus.cc:2229] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:07.550594 32458 raft_consensus.cc:2258] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.550698 32460 ts_tablet_manager.cc:1905] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.550897 32458 ts_tablet_manager.cc:1905] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.552245 32458 ts_tablet_manager.cc:1918] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:07.552312 32458 log.cc:1196] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/e910f11234fb44bebb85fe92764e5f8a
I20260926 05:42:07.552314 32460 ts_tablet_manager.cc:1918] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:07.552358 32460 log.cc:1196] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/e910f11234fb44bebb85fe92764e5f8a
I20260926 05:42:07.552531 32458 ts_tablet_manager.cc:1939] T e910f11234fb44bebb85fe92764e5f8a P 4128951d0e8e4051ac1597eda78a3747: Deleting consensus metadata
I20260926 05:42:07.552554 32460 ts_tablet_manager.cc:1939] T e910f11234fb44bebb85fe92764e5f8a P fea52a05957b41598459e22945bfab43: Deleting consensus metadata
I20260926 05:42:07.552906 32459 ts_tablet_manager.cc:1905] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.552991 31910 catalog_manager.cc:4841] TS 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813): tablet e910f11234fb44bebb85fe92764e5f8a (table test-table [id=b3e2d8137bea48c2be4884411bf47ec3]) successfully deleted
I20260926 05:42:07.553089 31910 catalog_manager.cc:4841] TS fea52a05957b41598459e22945bfab43 (127.23.213.194:45781): tablet e910f11234fb44bebb85fe92764e5f8a (table test-table [id=b3e2d8137bea48c2be4884411bf47ec3]) successfully deleted
I20260926 05:42:07.553123 31924 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51164:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:42:07.553251 31924 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:07.554359 32459 ts_tablet_manager.cc:1918] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:07.554438 32459 log.cc:1196] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/e910f11234fb44bebb85fe92764e5f8a
I20260926 05:42:07.554492 32061 tablet_service.cc:1467] Processing CreateTablet for tablet 1504cc25ef6f4ce6a92d90062689debd (DEFAULT_TABLE table=test-table [id=744b9722e5994b329eff8fc047bac290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:07.554585 32061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1504cc25ef6f4ce6a92d90062689debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.554649 32459 ts_tablet_manager.cc:1939] T e910f11234fb44bebb85fe92764e5f8a P 4cddb156244a4ab6b6833fa673046b05: Deleting consensus metadata
I20260926 05:42:07.554913 32341 tablet_service.cc:1467] Processing CreateTablet for tablet 1504cc25ef6f4ce6a92d90062689debd (DEFAULT_TABLE table=test-table [id=744b9722e5994b329eff8fc047bac290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:07.554935 31909 catalog_manager.cc:4841] TS 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611): tablet e910f11234fb44bebb85fe92764e5f8a (table test-table [id=b3e2d8137bea48c2be4884411bf47ec3]) successfully deleted
I20260926 05:42:07.554993 32341 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1504cc25ef6f4ce6a92d90062689debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.555716 32438 tablet_bootstrap.cc:492] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: Bootstrap starting.
I20260926 05:42:07.555753 32201 tablet_service.cc:1467] Processing CreateTablet for tablet 1504cc25ef6f4ce6a92d90062689debd (DEFAULT_TABLE table=test-table [id=744b9722e5994b329eff8fc047bac290]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:07.555840 32201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1504cc25ef6f4ce6a92d90062689debd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:07.556135 32435 tablet_bootstrap.cc:492] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: Bootstrap starting.
I20260926 05:42:07.556340 32438 tablet_bootstrap.cc:654] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.556424 32435 tablet_bootstrap.cc:654] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.556911 32436 tablet_bootstrap.cc:492] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: Bootstrap starting.
I20260926 05:42:07.556943 32438 tablet_bootstrap.cc:492] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: No bootstrap required, opened a new log
I20260926 05:42:07.556977 32438 ts_tablet_manager.cc:1397] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:07.557135 32438 raft_consensus.cc:348] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.557186 32436 tablet_bootstrap.cc:654] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:07.557219 32438 raft_consensus.cc:374] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.557243 32438 raft_consensus.cc:729] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4128951d0e8e4051ac1597eda78a3747, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.557303 32438 consensus_queue.cc:260] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.557451 32438 ts_tablet_manager.cc:1428] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.557668 32435 tablet_bootstrap.cc:492] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: No bootstrap required, opened a new log
I20260926 05:42:07.557718 32435 ts_tablet_manager.cc:1397] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:07.557749 32436 tablet_bootstrap.cc:492] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: No bootstrap required, opened a new log
I20260926 05:42:07.557819 32436 ts_tablet_manager.cc:1397] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:07.557866 32435 raft_consensus.cc:348] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.557936 32435 raft_consensus.cc:374] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.557934 32436 raft_consensus.cc:348] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.557962 32435 raft_consensus.cc:729] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4cddb156244a4ab6b6833fa673046b05, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.557981 32436 raft_consensus.cc:374] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:07.558018 32436 raft_consensus.cc:729] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fea52a05957b41598459e22945bfab43, State: Initialized, Role: FOLLOWER
I20260926 05:42:07.558008 32435 consensus_queue.cc:260] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.558071 32436 consensus_queue.cc:260] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.558125 32435 ts_tablet_manager.cc:1428] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.558161 32436 ts_tablet_manager.cc:1428] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:07.826957 32476 raft_consensus.cc:482] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:07.827078 32476 raft_consensus.cc:504] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.827260 32476 leader_election.cc:288] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), fea52a05957b41598459e22945bfab43 (127.23.213.194:45781)
I20260926 05:42:07.827502 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1504cc25ef6f4ce6a92d90062689debd" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747" is_pre_election: true
I20260926 05:42:07.827597 32081 raft_consensus.cc:2454] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 0.
I20260926 05:42:07.827594 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1504cc25ef6f4ce6a92d90062689debd" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea52a05957b41598459e22945bfab43" is_pre_election: true
I20260926 05:42:07.827729 32221 raft_consensus.cc:2454] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 0.
I20260926 05:42:07.827809 32294 leader_election.cc:302] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4128951d0e8e4051ac1597eda78a3747, 4cddb156244a4ab6b6833fa673046b05; no voters:
I20260926 05:42:07.827970 32476 raft_consensus.cc:2781] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:07.828050 32476 raft_consensus.cc:482] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:07.828071 32476 raft_consensus.cc:3037] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.828847 32476 raft_consensus.cc:504] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.829001 32476 leader_election.cc:288] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 election: Requested vote from peers 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813), fea52a05957b41598459e22945bfab43 (127.23.213.194:45781)
I20260926 05:42:07.829147 32081 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1504cc25ef6f4ce6a92d90062689debd" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4128951d0e8e4051ac1597eda78a3747"
I20260926 05:42:07.829171 32221 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1504cc25ef6f4ce6a92d90062689debd" candidate_uuid: "4cddb156244a4ab6b6833fa673046b05" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fea52a05957b41598459e22945bfab43"
I20260926 05:42:07.829231 32081 raft_consensus.cc:3037] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.829236 32221 raft_consensus.cc:3037] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:07.830016 32221 raft_consensus.cc:2454] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 1.
I20260926 05:42:07.830016 32081 raft_consensus.cc:2454] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4cddb156244a4ab6b6833fa673046b05 in term 1.
I20260926 05:42:07.830246 32294 leader_election.cc:302] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4cddb156244a4ab6b6833fa673046b05, fea52a05957b41598459e22945bfab43; no voters:
I20260926 05:42:07.830376 32476 raft_consensus.cc:2781] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:07.830493 32476 raft_consensus.cc:686] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 1 LEADER]: Becoming Leader. State: Replica: 4cddb156244a4ab6b6833fa673046b05, State: Running, Role: LEADER
I20260926 05:42:07.830590 32476 consensus_queue.cc:237] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } }
I20260926 05:42:07.831226 31922 catalog_manager.cc:5495] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195). New cstate: current_term: 1 leader_uuid: "4cddb156244a4ab6b6833fa673046b05" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4cddb156244a4ab6b6833fa673046b05" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38611 } health_report { overall_health: HEALTHY } } }
I20260926 05:42:07.844452 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.841081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54648 (local address 127.23.213.195:38611)
0926 05:42:07.841191 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.841193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.841200 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:42:07.841222 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.841222 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.841233 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.841298 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:07.841408 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.842417 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.843385 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.843687 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.843758 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.843963 (+ 205us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.843997 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.844070 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.844310 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.844335 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93,"thread_start_us":61,"threads_started":1}
I20260926 05:42:07.853212 32520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.847263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38611 (local address 127.23.213.195:38103)
0926 05:42:07.847495 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:07.847507 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:07.847566 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:07.850170 (+ 2604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:07.850178 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:07.850194 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 05:42:07.850274 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.850276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.851300 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.851301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.852265 (+ 964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:07.852267 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.852706 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.852707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:07.852770 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.852775 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.852978 (+ 203us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.852979 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:42:07.853063 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 05:42:07.853110 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"thread_start_us":79,"threads_started":1}
I20260926 05:42:07.853375 32517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:07.849895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:38103 (local address 127.23.213.195:38611)
0926 05:42:07.849950 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:07.849953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:07.849961 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:07.849987 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:07.849988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:07.849997 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:07.850062 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:07.850388 (+ 326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.851158 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.852353 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:07.852614 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:07.852689 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:07.852857 (+ 168us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:07.852891 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:07.852955 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:07.853313 (+ 358us) server_negotiation.cc:299] Negotiation successful
0926 05:42:07.853332 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:42:07.856186 32221 raft_consensus.cc:1261] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Refusing update from remote peer 4cddb156244a4ab6b6833fa673046b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.856230 32081 raft_consensus.cc:1261] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Refusing update from remote peer 4cddb156244a4ab6b6833fa673046b05: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:07.856581 32516 consensus_queue.cc:1035] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fea52a05957b41598459e22945bfab43" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 45781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.856799 32514 consensus_queue.cc:1035] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4128951d0e8e4051ac1597eda78a3747" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:07.871119 31922 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:07.871304 31922 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51164:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:07.872126 31922 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79a9cee23f1047019a5cb8a9b326d73b: Sending DeleteTablet for 3 replicas of tablet 1504cc25ef6f4ce6a92d90062689debd
I20260926 05:42:07.872464 32201 tablet_service.cc:1514] Processing DeleteTablet for tablet 1504cc25ef6f4ce6a92d90062689debd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:49600
I20260926 05:42:07.872479 32340 tablet_service.cc:1514] Processing DeleteTablet for tablet 1504cc25ef6f4ce6a92d90062689debd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43298
I20260926 05:42:07.872503 32061 tablet_service.cc:1514] Processing DeleteTablet for tablet 1504cc25ef6f4ce6a92d90062689debd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:07 UTC) from {username='oryx', principal='oryx/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42916
I20260926 05:42:07.872586 32460 tablet_replica.cc:330] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: stopping tablet replica
I20260926 05:42:07.872603 32459 tablet_replica.cc:330] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: stopping tablet replica
I20260926 05:42:07.872637 32460 raft_consensus.cc:2229] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:07.872660 32459 raft_consensus.cc:2229] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:42:07.872678 32460 raft_consensus.cc:2258] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.872762 32459 raft_consensus.cc:2258] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.872860 32458 tablet_replica.cc:330] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: stopping tablet replica
I20260926 05:42:07.872921 32458 raft_consensus.cc:2229] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:07.872972 32458 raft_consensus.cc:2258] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:07.873132 32460 ts_tablet_manager.cc:1905] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.873266 32459 ts_tablet_manager.cc:1905] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.873379 32458 ts_tablet_manager.cc:1905] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:07.874854 32458 ts_tablet_manager.cc:1918] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:42:07.874846 32460 ts_tablet_manager.cc:1918] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:42:07.874913 32458 log.cc:1196] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/1504cc25ef6f4ce6a92d90062689debd
I20260926 05:42:07.874913 32460 log.cc:1196] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/1504cc25ef6f4ce6a92d90062689debd
I20260926 05:42:07.875093 32459 ts_tablet_manager.cc:1918] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 05:42:07.875156 32460 ts_tablet_manager.cc:1939] T 1504cc25ef6f4ce6a92d90062689debd P fea52a05957b41598459e22945bfab43: Deleting consensus metadata
I20260926 05:42:07.875156 32458 ts_tablet_manager.cc:1939] T 1504cc25ef6f4ce6a92d90062689debd P 4128951d0e8e4051ac1597eda78a3747: Deleting consensus metadata
I20260926 05:42:07.875188 32459 log.cc:1196] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/1504cc25ef6f4ce6a92d90062689debd
I20260926 05:42:07.875600 31910 catalog_manager.cc:4841] TS 4128951d0e8e4051ac1597eda78a3747 (127.23.213.193:46813): tablet 1504cc25ef6f4ce6a92d90062689debd (table test-table [id=744b9722e5994b329eff8fc047bac290]) successfully deleted
I20260926 05:42:07.875659 32459 ts_tablet_manager.cc:1939] T 1504cc25ef6f4ce6a92d90062689debd P 4cddb156244a4ab6b6833fa673046b05: Deleting consensus metadata
I20260926 05:42:07.875679 31910 catalog_manager.cc:4841] TS fea52a05957b41598459e22945bfab43 (127.23.213.194:45781): tablet 1504cc25ef6f4ce6a92d90062689debd (table test-table [id=744b9722e5994b329eff8fc047bac290]) successfully deleted
I20260926 05:42:07.875902 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 31996
I20260926 05:42:07.876479 31909 catalog_manager.cc:4841] TS 4cddb156244a4ab6b6833fa673046b05 (127.23.213.195:38611): tablet 1504cc25ef6f4ce6a92d90062689debd (table test-table [id=744b9722e5994b329eff8fc047bac290]) successfully deleted
I20260926 05:42:07.883009 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 32136
I20260926 05:42:07.893224 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 32276
I20260926 05:42:07.904506 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 31892
2026-09-26T05:42:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3157 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[32528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[32528](info): set up 2 sockets
Sep 26 05:42:07 dist-test-slave-1fdt krb5kdc[32528](info): commencing operation
krb5kdc: starting...
W20260926 05:42:08.938478 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.011s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:42:09 dist-test-slave-1fdt krb5kdc[32528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401329, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:42:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:42:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:42:09.056298 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:34775
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46299
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:34775
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:09.137765 32541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:09.137979 32541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:09.138001 32541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:09.139142 32541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:09.139181 32541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:09.139195 32541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:09.139209 32541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:09.139221 32541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:09.140759 32541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46299
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:34775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:34775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:09.140971 32541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:09.141130 32541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:09.145978 32549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:09.145988 32547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:09.145979 32546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:09.146260 32541 server_base.cc:1034] running on GCE node
I20260926 05:42:09.146538 32541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:09.146819 32541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:09.147986 32541 hybrid_clock.cc:648] HybridClock initialized: now 1790401329147960 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T05:42:09Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1246 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:42:09 dist-test-slave-1fdt krb5kdc[32554](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:42:09 dist-test-slave-1fdt krb5kdc[32554](info): set up 2 sockets
Sep 26 05:42:09 dist-test-slave-1fdt krb5kdc[32554](info): commencing operation
krb5kdc: starting...
W20260926 05:42:10.194339 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:42:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:42:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:42:10.312083 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38421
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:10.394002 32567 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:10.394215 32567 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:10.394238 32567 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:10.395403 32567 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:10.395447 32567 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:10.395462 32567 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:10.395475 32567 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:10.395537 32567 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:10.397039 32567 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32567
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:10.397294 32567 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:10.397528 32567 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:10.402637 32575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.402614 32573 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.402614 32572 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:10.402642 32567 server_base.cc:1034] running on GCE node
I20260926 05:42:10.403069 32567 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:10.403358 32567 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:10.404556 32567 hybrid_clock.cc:648] HybridClock initialized: now 1790401330404535 us; error 47 us; skew 500 ppm
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:10.410777 32567 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:42:10.411459 32567 webserver.cc:460] Webserver started at http://127.23.213.254:34375/ using document root <none> and password file <none>
I20260926 05:42:10.411712 32567 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:10.411787 32567 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:10.411957 32567 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:10.413067 32567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "76af06b0575f4884a4c55729f53b9d90"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "9f4fb43ad0d2dab5dea03336216e0042"
server_key_iv: "09f9ca5a84f4cc705f0c7b9f8343ee11"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.413450 32567 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "76af06b0575f4884a4c55729f53b9d90"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "9f4fb43ad0d2dab5dea03336216e0042"
server_key_iv: "09f9ca5a84f4cc705f0c7b9f8343ee11"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.414922 32567 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:42:10.415809 32582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.416009 32567 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:10.416117 32567 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "76af06b0575f4884a4c55729f53b9d90"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "9f4fb43ad0d2dab5dea03336216e0042"
server_key_iv: "09f9ca5a84f4cc705f0c7b9f8343ee11"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.416208 32567 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:10.429996 32567 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:10.431895 32567 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:10.432072 32567 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:10.436269 32567 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38421
I20260926 05:42:10.436305 32634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38421 every 8 connection(s)
I20260926 05:42:10.436676 32567 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:42:10.437077 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 32567
I20260926 05:42:10.437201 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:42:10.437390 24407 external_mini_cluster.cc:1428] Setting key b5659e10faf8f09ff48a191c0b442a68
I20260926 05:42:10.437732 32635 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:10.441231 32635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Bootstrap starting.
I20260926 05:42:10.441987 32635 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:10.442293 32635 log.cc:824] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:10.443068 32635 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: No bootstrap required, opened a new log
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:10.445109 32635 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:10.445295 32635 raft_consensus.cc:374] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:10.445338 32635 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76af06b0575f4884a4c55729f53b9d90, State: Initialized, Role: FOLLOWER
I20260926 05:42:10.445477 32635 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:10.445564 32635 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:42:10.445622 32635 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:42:10.445674 32635 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:10.446471 32635 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:10.446583 32635 leader_election.cc:302] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76af06b0575f4884a4c55729f53b9d90; no voters:
I20260926 05:42:10.446740 32635 leader_election.cc:288] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:42:10.446791 32640 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:10.446938 32640 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 LEADER]: Becoming Leader. State: Replica: 76af06b0575f4884a4c55729f53b9d90, State: Running, Role: LEADER
I20260926 05:42:10.447017 32635 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:10.447064 32640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:10.447558 32641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "76af06b0575f4884a4c55729f53b9d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } } }
I20260926 05:42:10.447633 32641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:10.448081 32642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76af06b0575f4884a4c55729f53b9d90. Latest consensus state: current_term: 1 leader_uuid: "76af06b0575f4884a4c55729f53b9d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } } }
I20260926 05:42:10.448220 32642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:10.448364 32649 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:42:10.449087 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:10.437967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49404 (local address 127.23.213.254:38421)
0926 05:42:10.438167 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:10.438177 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:10.438211 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:42:10.438940 (+ 729us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:10.438950 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:10.439017 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:10.439179 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:10.439755 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.441107 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.442003 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.442615 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.442706 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:10.445013 (+ 2307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:10.445018 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:10.445018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:10.445031 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:10.447134 (+ 2103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.447601 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.447604 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.447604 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.447720 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.447947 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.447948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.447948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.448302 (+ 354us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:10.448695 (+ 393us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:10.448763 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 05:42:10.448925 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":91,"threads_started":1}
I20260926 05:42:10.449427 32649 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:42:10.450907 32649 catalog_manager.cc:1347] Generated new cluster ID: f442e4d5b932493fb51ef71efce74bb3
I20260926 05:42:10.450960 32649 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:42:10.459111 32649 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:42:10.459694 32649 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:42:10.463089 32649 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Generated new TSK 0
I20260926 05:42:10.463286 32649 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:42:10.496625 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38421
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:10.579929 32661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:10.580176 32661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:10.580214 32661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:10.581391 32661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:10.581465 32661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:10.581511 32661 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:42:10.583040 32661 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:10.583312 32661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:10.583556 32661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:10.588634 32667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.588634 32669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.588665 32666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:10.588871 32661 server_base.cc:1034] running on GCE node
I20260926 05:42:10.589097 32661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:10.589301 32661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:10.590438 32661 hybrid_clock.cc:648] HybridClock initialized: now 1790401330590427 us; error 27 us; skew 500 ppm
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:10.592984 32661 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:42:10.593367 32661 webserver.cc:460] Webserver started at http://127.23.213.193:42117/ using document root <none> and password file <none>
I20260926 05:42:10.593568 32661 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:10.593617 32661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:10.593703 32661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:10.594781 32661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "4e410f7c9e294a0595ec1efa1ef6842f"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "ba811f2b6c5d8f7c9d3a647ecd56418d"
server_key_iv: "d503e96227e80a077e1ae99dc06004cc"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.595084 32661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "4e410f7c9e294a0595ec1efa1ef6842f"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "ba811f2b6c5d8f7c9d3a647ecd56418d"
server_key_iv: "d503e96227e80a077e1ae99dc06004cc"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.596400 32661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:42:10.597182 32676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.597390 32661 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:10.597493 32661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "4e410f7c9e294a0595ec1efa1ef6842f"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "ba811f2b6c5d8f7c9d3a647ecd56418d"
server_key_iv: "d503e96227e80a077e1ae99dc06004cc"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.597576 32661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:10.607991 32661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:10.610765 32661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:10.610888 32661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:10.611145 32661 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:10.611421 32661 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:10.611452 32661 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.611475 32661 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:10.611488 32661 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.617106 32661 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:37411
I20260926 05:42:10.617204 321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:37411 every 8 connection(s)
I20260926 05:42:10.617515 32661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:42:10.622057 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 32661
I20260926 05:42:10.622190 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:42:10.622434 24407 external_mini_cluster.cc:1428] Setting key 90ab35014677a556b7104e54e77c6ba7
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:10.627919 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:10.618409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:57635 (local address 127.23.213.254:38421)
0926 05:42:10.618491 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:10.618494 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:10.618683 (+ 189us) server_negotiation.cc:407] Connection header received
0926 05:42:10.619259 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:10.619260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:10.619271 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:10.619338 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:10.620386 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.620944 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.622043 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.622399 (+ 356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.622495 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:10.624738 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:10.624739 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:10.624740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:10.624747 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:10.626200 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.626725 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.626727 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.626727 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.626801 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.627143 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.627144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.627144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.627318 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:10.627418 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:10.627772 (+ 354us) server_negotiation.cc:299] Negotiation successful
0926 05:42:10.627851 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 05:42:10.628216 324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:10.618366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:57635)
0926 05:42:10.618522 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:10.618565 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:10.619125 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:10.619414 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:10.619418 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:10.620066 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:10.620274 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:10.620280 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.621043 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.621043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:10.621936 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:10.621938 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.622517 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.622517 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:10.622674 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:10.623268 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:10.623277 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:10.624645 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:10.626281 (+ 1636us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:10.626288 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:10.626290 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:10.626613 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:10.626879 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:10.626880 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:10.626880 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:10.627048 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:10.627425 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:10.627433 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:10.627622 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 05:42:10.627748 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 05:42:10.627935 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260926 05:42:10.628706 322 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38421
I20260926 05:42:10.628829 322 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:10.629114 322 heartbeater.cc:511] Master 127.23.213.254:38421 requested a full tablet report, sending...
I20260926 05:42:10.629671 32599 ts_manager.cc:194] Registered new tserver with Master: 4e410f7c9e294a0595ec1efa1ef6842f (127.23.213.193:37411)
I20260926 05:42:10.630349 32599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:57635
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:42:10.649947 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38421
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:10.730249 327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:10.730499 327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:10.730537 327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:10.731683 327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:10.731745 327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:10.731788 327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:42:10.733207 327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:10.733469 327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:10.733700 327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:10.738915 332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.739027 335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.738874 333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:10.739626 327 server_base.cc:1034] running on GCE node
I20260926 05:42:10.739850 327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:10.740118 327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:10.741268 327 hybrid_clock.cc:648] HybridClock initialized: now 1790401330741250 us; error 34 us; skew 500 ppm
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:10.743803 327 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:42:10.744256 327 webserver.cc:460] Webserver started at http://127.23.213.194:43289/ using document root <none> and password file <none>
I20260926 05:42:10.744486 327 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:10.744549 327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:10.744694 327 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:10.745895 327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "3742bea5b3e64fa384d9da8d98ea206b"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "4c92f8f0decc399bcbd76f682ae76312"
server_key_iv: "f8ec1d51d02a3e05332886a6221eee36"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.746245 327 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "3742bea5b3e64fa384d9da8d98ea206b"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "4c92f8f0decc399bcbd76f682ae76312"
server_key_iv: "f8ec1d51d02a3e05332886a6221eee36"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.747577 327 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:10.748311 342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.748477 327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.748575 327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "3742bea5b3e64fa384d9da8d98ea206b"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "4c92f8f0decc399bcbd76f682ae76312"
server_key_iv: "f8ec1d51d02a3e05332886a6221eee36"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.748646 327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:10.765452 327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:10.767589 327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:10.767787 327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:10.768050 327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:10.768378 327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:10.768433 327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.768474 327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:10.768501 327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.774391 327 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:33691
I20260926 05:42:10.774435 455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:33691 every 8 connection(s)
I20260926 05:42:10.774858 327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:10.783601 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:10.775658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:59907 (local address 127.23.213.254:38421)
0926 05:42:10.775732 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:10.775734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:10.775895 (+ 161us) server_negotiation.cc:407] Connection header received
0926 05:42:10.776386 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:10.776388 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:10.776397 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:10.776455 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:10.777358 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.777911 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.778896 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.779083 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.779133 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:10.781189 (+ 2056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:10.781191 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:10.781191 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:10.781198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:10.782435 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.782794 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.782794 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.782795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.782842 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.783079 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.783079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.783080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.783181 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:10.783230 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:10.783464 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 05:42:10.783523 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:42:10.783838 458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:10.775606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:59907)
0926 05:42:10.775766 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:10.775808 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:10.776302 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:10.776511 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:10.776515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:10.777078 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:10.777278 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:10.777283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.778012 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.778013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:10.778818 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:10.778820 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.779157 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.779158 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:10.779258 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:10.779706 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:10.779714 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:10.781098 (+ 1384us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:10.782510 (+ 1412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:10.782516 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:10.782517 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:10.782730 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:10.782899 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:10.782899 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:10.782900 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:10.783025 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:10.783237 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:10.783241 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:10.783357 (+ 116us) client_negotiation.cc:769] Sending connection context
0926 05:42:10.783443 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:42:10.783575 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260926 05:42:10.784399 456 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38421
I20260926 05:42:10.784504 456 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:10.784682 456 heartbeater.cc:511] Master 127.23.213.254:38421 requested a full tablet report, sending...
I20260926 05:42:10.784978 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 327
I20260926 05:42:10.785111 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:42:10.785219 32599 ts_manager.cc:194] Registered new tserver with Master: 3742bea5b3e64fa384d9da8d98ea206b (127.23.213.194:33691)
I20260926 05:42:10.785367 24407 external_mini_cluster.cc:1428] Setting key 66b8d2daf4e613b1e1fd454200cd4938
I20260926 05:42:10.785686 32599 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:59907
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:42:10.813293 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38421
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:10.894261 461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:10.894479 461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:10.894502 461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:10.895609 461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:10.895644 461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:10.895670 461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:42:10.897100 461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:10.897312 461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:10.897552 461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:10.902557 469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.902568 466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:10.902619 467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:10.902653 461 server_base.cc:1034] running on GCE node
I20260926 05:42:10.902987 461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:10.903257 461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:10.904417 461 hybrid_clock.cc:648] HybridClock initialized: now 1790401330904400 us; error 46 us; skew 500 ppm
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:10.906829 461 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:42:10.907253 461 webserver.cc:460] Webserver started at http://127.23.213.195:45935/ using document root <none> and password file <none>
I20260926 05:42:10.907476 461 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:10.907543 461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:10.907696 461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:10.908740 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "e1c589b6b729424cb97661acd0f5dcc5"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "48a584e01a717df23593aab4b7d9c28f"
server_key_iv: "51a940c8bd945c4c35f5815bdc0578ee"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.909096 461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "e1c589b6b729424cb97661acd0f5dcc5"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "48a584e01a717df23593aab4b7d9c28f"
server_key_iv: "51a940c8bd945c4c35f5815bdc0578ee"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.910507 461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:42:10.911276 476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.911487 461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:10.911593 461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "e1c589b6b729424cb97661acd0f5dcc5"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "48a584e01a717df23593aab4b7d9c28f"
server_key_iv: "51a940c8bd945c4c35f5815bdc0578ee"
server_key_version: "encryptionkey@0"
I20260926 05:42:10.911664 461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:10.937580 461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:10.940007 461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:10.940130 461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:10.940390 461 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:10.940646 461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:10.940678 461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.940701 461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:10.940714 461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:10.946605 461 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:43197
I20260926 05:42:10.946694 589 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:43197 every 8 connection(s)
I20260926 05:42:10.947044 461 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:42:10.948338 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 461
I20260926 05:42:10.948453 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:42:10.948642 24407 external_mini_cluster.cc:1428] Setting key 628faeca305b57d81fb9809e9df3e8a5
Sep 26 05:42:10 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401330, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:10.956926 32638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:10.947792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:60095 (local address 127.23.213.254:38421)
0926 05:42:10.947860 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:10.947863 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:10.948108 (+ 245us) server_negotiation.cc:407] Connection header received
0926 05:42:10.948796 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:10.948798 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:10.948810 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:10.948884 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:10.950247 (+ 1363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.950753 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.951821 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.952026 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.952087 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:10.954372 (+ 2285us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:10.954373 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:10.954373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:10.954380 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:10.955692 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.956063 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.956064 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.956065 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.956122 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:10.956342 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:10.956342 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:10.956343 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:10.956451 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:10.956504 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:10.956759 (+ 255us) server_negotiation.cc:299] Negotiation successful
0926 05:42:10.956826 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 05:42:10.957044 592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:10.947798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:60095)
0926 05:42:10.947976 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:10.948013 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:10.948699 (+ 686us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:10.949276 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:10.949281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:10.949964 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:10.950166 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:10.950171 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.950830 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.950831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:10.951715 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:10.951717 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:10.952103 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:10.952104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:10.952202 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:10.952770 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:10.952778 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:10.954267 (+ 1489us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:10.955762 (+ 1495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:10.955768 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:10.955769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:10.955977 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:10.956189 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:10.956189 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:10.956190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:10.956283 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:10.956507 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:10.956511 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:10.956625 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 05:42:10.956707 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:42:10.956825 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":81,"threads_started":1}
I20260926 05:42:10.957515 590 heartbeater.cc:347] Connected to a master server at 127.23.213.254:38421
I20260926 05:42:10.957623 590 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:10.957844 590 heartbeater.cc:511] Master 127.23.213.254:38421 requested a full tablet report, sending...
I20260926 05:42:10.958272 32598 ts_manager.cc:194] Registered new tserver with Master: e1c589b6b729424cb97661acd0f5dcc5 (127.23.213.195:43197)
I20260926 05:42:10.958678 32598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:60095
I20260926 05:42:10.960304 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab.
Entry for principal oryx/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab.
WARNING: no policy specified for oryx/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab.
Entry for principal oryx/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab.
WARNING: no policy specified for oryx/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab.
Entry for principal oryx/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab.
I20260926 05:42:11.032490 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 32661
I20260926 05:42:11.036999 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 327
I20260926 05:42:11.041445 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 461
I20260926 05:42:11.045674 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 32567
I20260926 05:42:11.050745 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38421
--webserver_interface=127.23.213.254
--webserver_port=34375
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:11.132282 599 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:11.132539 599 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:11.132577 599 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:11.133771 599 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:11.133842 599 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:11.133867 599 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:11.133896 599 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:11.133919 599 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:11.135430 599 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=34375
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.599
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:11.135712 599 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:11.135953 599 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:11.141013 605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:11.141029 604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:11.141060 599 server_base.cc:1034] running on GCE node
W20260926 05:42:11.141042 607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:11.141561 599 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:11.141965 599 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:11.143117 599 hybrid_clock.cc:648] HybridClock initialized: now 1790401331143095 us; error 38 us; skew 500 ppm
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401331, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:11.145964 599 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:42:11.146461 599 webserver.cc:460] Webserver started at http://127.23.213.254:34375/ using document root <none> and password file <none>
I20260926 05:42:11.146706 599 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:11.146795 599 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:11.148154 599 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:11.148806 614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.149016 599 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:11.149122 599 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "76af06b0575f4884a4c55729f53b9d90"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "9f4fb43ad0d2dab5dea03336216e0042"
server_key_iv: "09f9ca5a84f4cc705f0c7b9f8343ee11"
server_key_version: "encryptionkey@0"
I20260926 05:42:11.149487 599 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:11.162695 599 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:11.165123 599 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:11.165297 599 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:11.169333 599 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38421
I20260926 05:42:11.169371 666 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38421 every 8 connection(s)
I20260926 05:42:11.169767 599 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:42:11.171219 667 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:42:11.171891 667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Bootstrap starting.
I20260926 05:42:11.175235 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 599
I20260926 05:42:11.175549 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:37411
--local_ip_for_outbound_sockets=127.23.213.193
--tserver_master_addrs=127.23.213.254:38421
--webserver_port=42117
--webserver_interface=127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:42:11.180146 667 log.cc:824] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:11.181963 667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:42:11.182255 667 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Bootstrap complete.
I20260926 05:42:11.184530 667 raft_consensus.cc:348] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:11.184778 667 raft_consensus.cc:729] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76af06b0575f4884a4c55729f53b9d90, State: Initialized, Role: FOLLOWER
I20260926 05:42:11.185056 667 consensus_queue.cc:260] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:11.185160 667 raft_consensus.cc:388] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:42:11.185214 667 raft_consensus.cc:482] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:42:11.185264 667 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:42:11.186291 667 raft_consensus.cc:504] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:11.186404 667 leader_election.cc:302] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 76af06b0575f4884a4c55729f53b9d90; no voters:
I20260926 05:42:11.186599 667 leader_election.cc:288] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:42:11.186633 671 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:42:11.186743 671 raft_consensus.cc:686] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [term 2 LEADER]: Becoming Leader. State: Replica: 76af06b0575f4884a4c55729f53b9d90, State: Running, Role: LEADER
I20260926 05:42:11.186827 671 consensus_queue.cc:237] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } }
I20260926 05:42:11.186898 667 sys_catalog.cc:564] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:11.187085 672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "76af06b0575f4884a4c55729f53b9d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } } }
I20260926 05:42:11.187165 672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:11.187096 673 sys_catalog.cc:455] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 76af06b0575f4884a4c55729f53b9d90. Latest consensus state: current_term: 2 leader_uuid: "76af06b0575f4884a4c55729f53b9d90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "76af06b0575f4884a4c55729f53b9d90" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38421 } } }
I20260926 05:42:11.187314 673 sys_catalog.cc:458] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:11.187916 679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:42:11.188408 679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:42:11.188524 679 catalog_manager.cc:1259] Loaded cluster ID: f442e4d5b932493fb51ef71efce74bb3
I20260926 05:42:11.188548 679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:42:11.189949 679 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:42:11.190111 679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 76af06b0575f4884a4c55729f53b9d90: Loaded TSK: 0
I20260926 05:42:11.190308 679 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:42:11.258730 669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:11.258945 669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:11.258970 669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:11.260208 669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:11.260250 669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:11.260277 669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:42:11.261906 669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.193.keytab
--principal=oryx/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:37411
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=42117
--tserver_master_addrs=127.23.213.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:11.262138 669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:11.262396 669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:11.267519 696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:11.267463 694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:11.267475 693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:11.267970 669 server_base.cc:1034] running on GCE node
I20260926 05:42:11.268149 669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:11.268371 669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:11.269536 669 hybrid_clock.cc:648] HybridClock initialized: now 1790401331269515 us; error 34 us; skew 500 ppm
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401331, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:11.272156 669 init.cc:377] Logged in from keytab as oryx/127.23.213.193@KRBTEST.COM (short username oryx)
I20260926 05:42:11.272686 669 webserver.cc:460] Webserver started at http://127.23.213.193:42117/ using document root <none> and password file <none>
I20260926 05:42:11.272938 669 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:11.272995 669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:11.274600 669 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:42:11.275339 703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.275503 669 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:11.275584 669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "4e410f7c9e294a0595ec1efa1ef6842f"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "ba811f2b6c5d8f7c9d3a647ecd56418d"
server_key_iv: "d503e96227e80a077e1ae99dc06004cc"
server_key_version: "encryptionkey@0"
I20260926 05:42:11.275936 669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:11.300449 669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:11.303010 669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:11.303185 669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:11.303479 669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:11.303901 669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:11.303959 669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.304000 669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:11.304028 669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.311106 669 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:37411
I20260926 05:42:11.311153 816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:37411 every 8 connection(s)
I20260926 05:42:11.311483 669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.319880 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.312180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:44397)
0926 05:42:11.312353 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.312407 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.312917 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.313391 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.313395 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.314021 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.314220 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.314225 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.315367 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.315369 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.316167 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.316169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.316797 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.316798 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.316940 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.317488 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.317497 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.319657 (+ 2160us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":89,"threads_started":1}
W20260926 05:42:11.319917 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.312228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:44397 (local address 127.23.213.254:38421)
0926 05:42:11.312439 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.312450 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.312496 (+ 46us) server_negotiation.cc:407] Connection header received
0926 05:42:11.313129 (+ 633us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.313137 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.313187 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.313344 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.314350 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.315253 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.316281 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.316701 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.316823 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.319771 (+ 2948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:44397: BlockingRecv error: recv got EOF from 127.23.213.193:44397 (error 108)
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":91,"threads_started":1}
W20260926 05:42:11.320155 817 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:38421 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.213.254:38421: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
I20260926 05:42:11.320814 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 669
I20260926 05:42:11.321113 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:33691
--local_ip_for_outbound_sockets=127.23.213.194
--tserver_master_addrs=127.23.213.254:38421
--webserver_port=43289
--webserver_interface=127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.324898 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.320406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:60599)
0926 05:42:11.320438 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.320448 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.320512 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.320627 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.320627 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.320758 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.320798 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.320799 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.321591 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.321592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.322159 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.322161 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.322552 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.322553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.322618 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.323059 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.323064 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.324817 (+ 1753us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 05:42:11.324949 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.320454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:60599 (local address 127.23.213.254:38421)
0926 05:42:11.320510 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.320512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.320521 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:11.320568 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.320569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.320580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.320626 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.320900 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.321504 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.322255 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.322470 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.322542 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.324890 (+ 2348us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:60599: BlockingRecv error: recv got EOF from 127.23.213.193:60599 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.329228 819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.325185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:38957)
0926 05:42:11.325263 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.325274 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.325322 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.325447 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.325448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.325581 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.325606 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.325608 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.326273 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.326274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.326763 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.326764 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.327120 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.327121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.327172 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.327594 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.327599 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.329151 (+ 1552us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":63}
W20260926 05:42:11.329340 817 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:42:11.329281 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.325223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:38957 (local address 127.23.213.254:38421)
0926 05:42:11.325278 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.325280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.325308 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:42:11.325380 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.325381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.325390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.325462 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.325681 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.326192 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.326853 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.327051 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.327122 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.329221 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:38957: BlockingRecv error: recv got EOF from 127.23.213.193:38957 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 05:42:11.401140 821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:11.401336 821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:11.401360 821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:11.402550 821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:11.402590 821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:11.402616 821 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:42:11.404028 821 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.194.keytab
--principal=oryx/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:33691
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=43289
--tserver_master_addrs=127.23.213.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.821
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:11.404264 821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:11.404493 821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:11.409483 827 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:11.409531 821 server_base.cc:1034] running on GCE node
W20260926 05:42:11.409502 826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:11.409653 829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:11.409884 821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:11.410137 821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:11.411295 821 hybrid_clock.cc:648] HybridClock initialized: now 1790401331411276 us; error 41 us; skew 500 ppm
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401331, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:11.413714 821 init.cc:377] Logged in from keytab as oryx/127.23.213.194@KRBTEST.COM (short username oryx)
I20260926 05:42:11.414191 821 webserver.cc:460] Webserver started at http://127.23.213.194:43289/ using document root <none> and password file <none>
I20260926 05:42:11.414438 821 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:11.414515 821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:11.415853 821 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:42:11.416523 836 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.416697 821 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:42:11.416790 821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "3742bea5b3e64fa384d9da8d98ea206b"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "4c92f8f0decc399bcbd76f682ae76312"
server_key_iv: "f8ec1d51d02a3e05332886a6221eee36"
server_key_version: "encryptionkey@0"
I20260926 05:42:11.417194 821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:11.436455 821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:11.439036 821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:11.439185 821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:11.439469 821 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:11.439836 821 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:11.439898 821 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.439944 821 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:11.439976 821 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.446414 821 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:33691
I20260926 05:42:11.446453 949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:33691 every 8 connection(s)
I20260926 05:42:11.446849 821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.454330 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.447559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:35865)
0926 05:42:11.447725 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.447767 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.448235 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.448549 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.448560 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.449117 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.449309 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.449314 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.450072 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.450073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.450862 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.450863 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.451202 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.451203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.451302 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.451915 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.451923 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.454191 (+ 2268us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":91,"threads_started":1}
W20260926 05:42:11.454360 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.447581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:35865 (local address 127.23.213.254:38421)
0926 05:42:11.447625 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.447627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.447843 (+ 216us) server_negotiation.cc:407] Connection header received
0926 05:42:11.448353 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.448354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.448368 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.448449 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.449403 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.449993 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.450942 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.451131 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.451191 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.454308 (+ 3117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:35865: BlockingRecv error: recv got EOF from 127.23.213.194:35865 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 05:42:11.454596 950 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:38421 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.213.254:38421: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
I20260926 05:42:11.456068 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 821
I20260926 05:42:11.456370 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:43197
--local_ip_for_outbound_sockets=127.23.213.195
--tserver_master_addrs=127.23.213.254:38421
--webserver_port=45935
--webserver_interface=127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44491
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.459651 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.454897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:36645)
0926 05:42:11.454958 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.454970 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.455049 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.455201 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.455202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.455324 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.455364 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.455365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.455996 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.455997 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.456664 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.456665 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.457012 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.457013 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.457084 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.457533 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.457540 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.459549 (+ 2009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 05:42:11.459667 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.454903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:36645 (local address 127.23.213.254:38421)
0926 05:42:11.454964 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.454966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.455024 (+ 58us) server_negotiation.cc:407] Connection header received
0926 05:42:11.455118 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.455119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.455128 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.455199 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.455444 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.455917 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.456750 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.456935 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.457000 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.459623 (+ 2623us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:36645: BlockingRecv error: recv got EOF from 127.23.213.194:36645 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.464861 952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.459982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:36417)
0926 05:42:11.460096 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.460107 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.460143 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.460265 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.460266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.460417 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.460456 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.460457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.461107 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.461108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.462018 (+ 910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.462021 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.462350 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.462351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.462421 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.462837 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.462842 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.464763 (+ 1921us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96}
W20260926 05:42:11.464983 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.460051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:36417 (local address 127.23.213.254:38421)
0926 05:42:11.460124 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.460126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.460154 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:42:11.460205 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.460205 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.460214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.460260 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.460528 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.461025 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.462097 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.462274 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.462338 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.464887 (+ 2549us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:36417: BlockingRecv error: recv got EOF from 127.23.213.194:36417 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
W20260926 05:42:11.464988 950 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:42:11.539067 953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:11.539265 953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:11.539288 953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:11.540465 953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:11.540509 953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:11.540539 953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:42:11.542006 953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44491
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/krb5kdc/oryx_127.23.213.195.keytab
--principal=oryx/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:43197
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=45935
--tserver_master_addrs=127.23.213.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:11.542253 953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:11.542469 953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:11.547392 961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:11.547397 958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:11.547472 953 server_base.cc:1034] running on GCE node
W20260926 05:42:11.547399 959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:11.547869 953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:11.548135 953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:11.549327 953 hybrid_clock.cc:648] HybridClock initialized: now 1790401331549309 us; error 30 us; skew 500 ppm
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401331, etypes {rep=17 tkt=17 ses=17}, oryx/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:11.552285 953 init.cc:377] Logged in from keytab as oryx/127.23.213.195@KRBTEST.COM (short username oryx)
I20260926 05:42:11.552704 953 webserver.cc:460] Webserver started at http://127.23.213.195:45935/ using document root <none> and password file <none>
I20260926 05:42:11.552932 953 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:11.552990 953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:11.554422 953 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:11.555006 968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.555177 953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.555261 953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "e1c589b6b729424cb97661acd0f5dcc5"
format_stamp: "Formatted at 2026-09-26 05:42:10 on dist-test-slave-1fdt"
server_key: "48a584e01a717df23593aab4b7d9c28f"
server_key_iv: "51a940c8bd945c4c35f5815bdc0578ee"
server_key_version: "encryptionkey@0"
I20260926 05:42:11.555576 953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:11.575366 953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:11.578029 953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:11.578142 953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:11.578322 953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:11.578646 953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:11.578679 953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.578702 953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:11.578747 953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:11.584533 953 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:43197
I20260926 05:42:11.584619 1081 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:43197 every 8 connection(s)
I20260926 05:42:11.584940 953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.591848 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 953
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
W20260926 05:42:11.592881 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.585821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:40193 (local address 127.23.213.254:38421)
0926 05:42:11.585899 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.585901 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.586173 (+ 272us) server_negotiation.cc:407] Connection header received
0926 05:42:11.586780 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.586781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.586797 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.586919 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.588005 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.588542 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.589520 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.589711 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.589867 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.592823 (+ 2956us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:40193: BlockingRecv error: recv got EOF from 127.23.213.195:40193 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:42:11.593027 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.585868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:40193)
0926 05:42:11.586053 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.586089 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.586644 (+ 555us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.586984 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.586987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.587630 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.587916 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.587921 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.588622 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.588623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.589411 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.589412 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.589878 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.589879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.589980 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.590469 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.590477 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.592759 (+ 2282us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":106,"threads_started":1}
W20260926 05:42:11.593320 1082 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:38421 (0 consecutive failures): Not authorized: Failed to ping master at 127.23.213.254:38421: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.598368 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.593539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:44809)
0926 05:42:11.593633 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.593643 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.593691 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.593859 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.593860 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.593992 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.594048 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.594049 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.594785 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.594786 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.595374 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.595377 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.595816 (+ 439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.595818 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.595900 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.596336 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.596341 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.598292 (+ 1951us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":76}
W20260926 05:42:11.598671 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.593591 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:44809 (local address 127.23.213.254:38421)
0926 05:42:11.593634 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.593636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.593678 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:42:11.593780 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.593781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.593789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.593852 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.594160 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.594667 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.595495 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.595716 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.595824 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.598598 (+ 2774us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:44809: BlockingRecv error: recv got EOF from 127.23.213.195:44809 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:42:11.600080 1086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.592413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49420 (local address 127.23.213.254:38421)
0926 05:42:11.592564 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.592567 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.592574 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:42:11.592601 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.592601 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.592610 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.592681 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.593366 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.594193 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.594958 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.595172 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.595245 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.596575 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:11.596580 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:11.596580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:11.596593 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:11.598646 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:11.599052 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:11.599056 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:11.599056 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:11.599161 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:11.599367 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:11.599368 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:11.599368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:11.599601 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:11.599751 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:11.599862 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 05:42:11.599970 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":79,"threads_started":1}
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:11 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:11.603127 1084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:11.598791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:56719)
0926 05:42:11.598978 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:11.598987 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:11.599036 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:11.599170 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:11.599171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:11.599295 (+ 124us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:11.599321 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.599322 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.599930 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.599931 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.600666 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:11.600668 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.601045 (+ 377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.601046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:11.601099 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.601519 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:11.601524 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:11.603085 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154}
W20260926 05:42:11.603248 1082 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:42:11.603263 820 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:11.598835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:56719 (local address 127.23.213.254:38421)
0926 05:42:11.598876 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:11.598878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:11.599025 (+ 147us) server_negotiation.cc:407] Connection header received
0926 05:42:11.599100 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:11.599101 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:11.599110 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:11.599165 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:11.599399 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.599849 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.600734 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:11.600987 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:11.601052 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:11.603192 (+ 2140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:56719: BlockingRecv error: recv got EOF from 127.23.213.195:56719 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 26 05:42:12 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:12 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:12.334821 1087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:12.329933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:37261)
0926 05:42:12.330082 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:12.330095 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:12.330164 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:12.330372 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:12.330373 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:12.330554 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:12.330605 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:12.330607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.331250 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.331251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:12.331809 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:12.331811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.332133 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.332133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:12.332174 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:12.332729 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:12.332735 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:12.334756 (+ 2021us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":87,"threads_started":1}
W20260926 05:42:12.335018 1088 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:12.330049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:37261 (local address 127.23.213.254:38421)
0926 05:42:12.330190 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:12.330192 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:12.330200 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:12.330223 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:12.330223 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:12.330234 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:12.330305 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:12.330675 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.331174 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.331887 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.332074 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.332119 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:12.334926 (+ 2807us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:37261: BlockingRecv error: recv got EOF from 127.23.213.193:37261 (error 108)
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
Sep 26 05:42:12 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:12 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:12.469928 1089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:12.465326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:51231)
0926 05:42:12.465460 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:12.465469 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:12.465528 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:12.465655 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:12.465656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:12.465859 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:12.465907 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:12.465909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.466573 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.466574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:12.467101 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:12.467102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.467426 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.467427 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:12.467467 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:12.467985 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:12.467989 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:12.469857 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
W20260926 05:42:12.470019 1088 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:12.465408 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:51231 (local address 127.23.213.254:38421)
0926 05:42:12.465465 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:12.465467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:12.465524 (+ 57us) server_negotiation.cc:407] Connection header received
0926 05:42:12.465584 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:12.465585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:12.465594 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:12.465635 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:12.465986 (+ 351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.466504 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.467170 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.467350 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.467415 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:12.469935 (+ 2520us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:51231: BlockingRecv error: recv got EOF from 127.23.213.194:51231 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 05:42:12 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:12 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:12.608527 1090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:12.603657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:52751)
0926 05:42:12.603888 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:12.603898 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:12.603989 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:12.604179 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:12.604180 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:12.604363 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:12.604410 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:12.604412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.605102 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.605103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:12.605652 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:12.605654 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.606053 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.606054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:12.606137 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:12.606665 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:12.606670 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:12.608464 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":84,"threads_started":1}
W20260926 05:42:12.608647 1088 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:12.603751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:52751 (local address 127.23.213.254:38421)
0926 05:42:12.603836 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:12.603839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:12.603970 (+ 131us) server_negotiation.cc:407] Connection header received
0926 05:42:12.604082 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:12.604083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:12.604097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:12.604172 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:12.604494 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.605013 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.605766 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:12.605967 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:12.606053 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:12.608551 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:52751: BlockingRecv error: recv got EOF from 127.23.213.195:52751 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 05:42:13 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:13 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:13.339879 1091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:13.335248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:57721)
0926 05:42:13.335440 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:13.335451 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:13.335521 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:13.335647 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:13.335648 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:13.335802 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:13.335883 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:13.335885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.336638 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.336639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:13.337189 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:13.337190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.337512 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.337513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:13.337554 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:13.337992 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:13.337997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:13.339789 (+ 1792us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":101,"threads_started":1}
W20260926 05:42:13.339928 1092 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:13.335353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:57721 (local address 127.23.213.254:38421)
0926 05:42:13.335476 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:13.335478 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:13.335503 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:42:13.335577 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:13.335578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:13.335588 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:13.335637 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:13.335940 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.336557 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.337261 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.337447 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.337495 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:13.339875 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:57721: BlockingRecv error: recv got EOF from 127.23.213.193:57721 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
Sep 26 05:42:13 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:13 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:13.475736 1093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:13.470448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:38935)
0926 05:42:13.470653 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:13.470668 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:13.470771 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:13.470980 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:13.470981 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:13.471175 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:13.471228 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:13.471230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.471980 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.471981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:13.472544 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:13.472546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.472963 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.472964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:13.473036 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:13.473518 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:13.473523 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:13.475631 (+ 2108us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":99,"threads_started":1}
W20260926 05:42:13.475787 1092 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:13.470578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:38935 (local address 127.23.213.254:38421)
0926 05:42:13.470657 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:13.470660 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:13.470750 (+ 90us) server_negotiation.cc:407] Connection header received
0926 05:42:13.470862 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:13.470863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:13.470878 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:13.470969 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:13.471327 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.471879 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.472643 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.472862 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.472960 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:13.475711 (+ 2751us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:38935: BlockingRecv error: recv got EOF from 127.23.213.194:38935 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 05:42:13 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:13 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:13.613461 1094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:13.609026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:59073)
0926 05:42:13.609175 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:13.609189 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:13.609262 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:13.609391 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:13.609391 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:13.609551 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:13.609600 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:13.609602 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.610290 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.610291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:13.610824 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:13.610825 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.611172 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.611173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:13.611217 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:13.611628 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:13.611632 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:13.613405 (+ 1773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":69,"threads_started":1}
W20260926 05:42:13.613610 1092 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:13.609164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:59073 (local address 127.23.213.254:38421)
0926 05:42:13.609208 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:13.609210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:13.609247 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:42:13.609320 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:13.609320 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:13.609332 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:13.609371 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:13.609668 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.610216 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.610931 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:13.611108 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:13.611158 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:13.613519 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:59073: BlockingRecv error: recv got EOF from 127.23.213.195:59073 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 05:42:14 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:14 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:14.345094 1095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:14.340414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:52379)
0926 05:42:14.340562 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:14.340577 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:14.340644 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:14.340883 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:14.340884 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:14.341057 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:14.341112 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:14.341114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.341847 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.341848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:14.342385 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:14.342387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.342731 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.342732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:14.342774 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:14.343234 (+ 460us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:14.343238 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:14.345026 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
W20260926 05:42:14.345203 1096 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:14.340478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:52379 (local address 127.23.213.254:38421)
0926 05:42:14.340610 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:14.340612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:14.340622 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:14.340718 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:14.340719 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:14.340729 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:14.340788 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:14.341185 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.341684 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.342458 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.342657 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.342712 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:14.345126 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:52379: BlockingRecv error: recv got EOF from 127.23.213.193:52379 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":89,"threads_started":1}
Sep 26 05:42:14 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:14 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:14.480784 1097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:14.476220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:55635)
0926 05:42:14.476382 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:14.476392 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:14.476451 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:14.476594 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:14.476595 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:14.476758 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:14.476860 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:14.476862 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.477487 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.477488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:14.478077 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:14.478079 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.478397 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.478398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:14.478441 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:14.478868 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:14.478873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:14.480673 (+ 1800us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":100,"threads_started":1}
W20260926 05:42:14.480896 1096 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:14.476293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:55635 (local address 127.23.213.254:38421)
0926 05:42:14.476368 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:14.476370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:14.476452 (+ 82us) server_negotiation.cc:407] Connection header received
0926 05:42:14.476499 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:14.476500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:14.476509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:14.476574 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:14.476933 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.477413 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.478144 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.478330 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.478384 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:14.480805 (+ 2421us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:55635: BlockingRecv error: recv got EOF from 127.23.213.194:55635 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 05:42:14 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:14 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:14.619027 1098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:14.613996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:38423)
0926 05:42:14.614289 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:14.614302 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:14.614392 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:14.614550 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:14.614550 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:14.614728 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:14.614774 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:14.614775 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.615559 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.615560 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:14.616102 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:14.616103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.616478 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.616479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:14.616545 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:14.617054 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:14.617058 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:14.618956 (+ 1898us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":271,"thread_start_us":201,"threads_started":1}
W20260926 05:42:14.619100 1096 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:14.614211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:38423 (local address 127.23.213.254:38421)
0926 05:42:14.614296 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:14.614299 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:14.614371 (+ 72us) server_negotiation.cc:407] Connection header received
0926 05:42:14.614455 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:14.614456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:14.614468 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:14.614544 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:14.614904 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.615430 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.616186 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:14.616392 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:14.616477 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:14.619051 (+ 2574us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:38423: BlockingRecv error: recv got EOF from 127.23.213.195:38423 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Sep 26 05:42:15 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:15 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:15.350214 1099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:15.345671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:53935)
0926 05:42:15.345895 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:15.345906 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:15.345969 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:15.346133 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:15.346134 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:15.346284 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:15.346333 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:15.346334 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.346963 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.346964 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:15.347498 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:15.347499 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.347848 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.347849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:15.347894 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:15.348335 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:15.348340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:15.350154 (+ 1814us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":205,"thread_start_us":139,"threads_started":1}
W20260926 05:42:15.350368 1100 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:15.345805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:53935 (local address 127.23.213.254:38421)
0926 05:42:15.345915 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:15.345917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:15.345952 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:42:15.346041 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:15.346042 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:15.346051 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:15.346108 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:15.346397 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.346888 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.347597 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.347781 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.347831 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:15.350268 (+ 2437us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:53935: BlockingRecv error: recv got EOF from 127.23.213.193:53935 (error 108)
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":52,"threads_started":1}
Sep 26 05:42:15 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:15 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:15.489176 1101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:15.481368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:40037)
0926 05:42:15.481535 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:15.481546 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:15.481615 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:15.481766 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:15.481766 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:15.481921 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:15.481971 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:15.481972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.482700 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.482702 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:15.483266 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:15.483268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.483676 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.483677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:15.483733 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:15.487183 (+ 3450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:15.487188 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:15.489078 (+ 1890us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
W20260926 05:42:15.489246 1100 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:15.481420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:40037 (local address 127.23.213.254:38421)
0926 05:42:15.481500 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:15.481503 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:15.481616 (+ 113us) server_negotiation.cc:407] Connection header received
0926 05:42:15.481666 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:15.481667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:15.481679 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:15.481735 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:15.482055 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.482578 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.483384 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.483600 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.483677 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:15.489183 (+ 5506us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:40037: BlockingRecv error: recv got EOF from 127.23.213.194:40037 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 05:42:15 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:15 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:15.624740 1102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:15.619597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:45013)
0926 05:42:15.619740 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:15.619751 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:15.619816 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:15.619943 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:15.619943 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:15.620119 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:15.620167 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:15.620169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.621010 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.621011 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:15.621720 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:15.621723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.622198 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.622199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:15.622245 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:15.622752 (+ 507us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:15.622756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:15.624634 (+ 1878us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
W20260926 05:42:15.624830 1100 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:15.619682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:45013 (local address 127.23.213.254:38421)
0926 05:42:15.619733 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:15.619735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:15.619823 (+ 88us) server_negotiation.cc:407] Connection header received
0926 05:42:15.619859 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:15.619860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:15.619875 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:15.619926 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:15.620267 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.620918 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.621902 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:15.622124 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:15.622182 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:15.624738 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:45013: BlockingRecv error: recv got EOF from 127.23.213.195:45013 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 05:42:16 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:16 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:16.355506 1103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:16.350682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:59213)
0926 05:42:16.350838 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:16.350850 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:16.350917 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:16.351084 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:16.351085 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:16.351291 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:16.351377 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:16.351379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.352199 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.352200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:16.352795 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:16.352798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.353196 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.353197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:16.353248 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:16.353675 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:16.353679 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:16.355448 (+ 1769us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":99,"threads_started":1}
W20260926 05:42:16.355633 1104 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:16.350774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:59213 (local address 127.23.213.254:38421)
0926 05:42:16.350904 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:16.350906 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:16.350915 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:16.350968 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:16.350968 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:16.350978 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:16.351064 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:16.351458 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.352104 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.352898 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.353122 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.353178 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:16.355552 (+ 2374us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:59213: BlockingRecv error: recv got EOF from 127.23.213.193:59213 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":77,"threads_started":1}
Sep 26 05:42:16 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:16 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:16.494488 1105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:16.489673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:49947)
0926 05:42:16.489813 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:16.489832 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:16.489909 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:16.490062 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:16.490063 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:16.490237 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:16.490282 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:16.490284 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.490943 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.490944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:16.491464 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:16.491466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.491855 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.491856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:16.491923 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:16.492414 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:16.492419 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:16.494429 (+ 2010us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":57,"threads_started":1}
W20260926 05:42:16.494596 1104 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:16.489842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:49947 (local address 127.23.213.254:38421)
0926 05:42:16.489919 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:16.489921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:16.489931 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:42:16.489965 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:16.489966 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:16.489980 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:16.490056 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:16.490367 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.490858 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.491579 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.491771 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.491855 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:16.494524 (+ 2669us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:49947: BlockingRecv error: recv got EOF from 127.23.213.194:49947 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 05:42:16 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:16 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:16.629894 1106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:16.625267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:41553)
0926 05:42:16.625378 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:16.625389 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:16.625452 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:16.625592 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:16.625593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:16.625767 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:16.625817 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:16.625818 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.626481 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.626483 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:16.627055 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:16.627057 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.627393 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.627394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:16.627446 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:16.627923 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:16.627928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:16.629836 (+ 1908us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":55,"threads_started":1}
W20260926 05:42:16.630064 1104 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:16.625340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:41553 (local address 127.23.213.254:38421)
0926 05:42:16.625424 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:16.625426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:16.625435 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:16.625512 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:16.625513 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:16.625525 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:16.625577 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:16.625885 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.626383 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.627140 (+ 757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:16.627318 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:16.627375 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:16.629989 (+ 2614us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:41553: BlockingRecv error: recv got EOF from 127.23.213.195:41553 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Sep 26 05:42:17 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:17 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:17.360689 1107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:17.356027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:33917)
0926 05:42:17.356175 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:17.356185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:17.356248 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:17.356376 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:17.356377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:17.356529 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:17.356578 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:17.356580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.357209 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.357210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:17.357780 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:17.357781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.358125 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.358126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:17.358167 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:17.358752 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:17.358761 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:17.360629 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":86,"threads_started":1}
W20260926 05:42:17.360809 1108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:17.356109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:33917 (local address 127.23.213.254:38421)
0926 05:42:17.356233 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:17.356235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:17.356243 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:17.356301 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:17.356301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:17.356311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:17.356360 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:17.356648 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.357140 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.357845 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.358054 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.358105 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:17.360726 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:33917: BlockingRecv error: recv got EOF from 127.23.213.193:33917 (error 108)
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":66,"threads_started":1}
Sep 26 05:42:17 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:17 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:17.500053 1109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:17.495061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:36935)
0926 05:42:17.495209 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:17.495220 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:17.495273 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:17.495440 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:17.495441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:17.495607 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:17.495655 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:17.495657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.496607 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.496608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:17.497284 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:17.497286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.497627 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.497628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:17.497673 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:17.498251 (+ 578us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:17.498256 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:17.499992 (+ 1736us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":88,"threads_started":1}
W20260926 05:42:17.500299 1108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:17.495176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:36935 (local address 127.23.213.254:38421)
0926 05:42:17.495246 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:17.495248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:17.495261 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:42:17.495348 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:17.495348 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:17.495359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:17.495424 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:17.495716 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.496474 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.497360 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.497552 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.497607 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:17.500208 (+ 2601us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:36935: BlockingRecv error: recv got EOF from 127.23.213.194:36935 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 05:42:17 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:17 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:17.635105 1110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:17.630373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:52129)
0926 05:42:17.630529 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:17.630543 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:17.630630 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:17.630775 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:17.630776 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:17.630954 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:17.631005 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:17.631007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.631642 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.631643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:17.632198 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:17.632200 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.632552 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.632553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:17.632608 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:17.633111 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:17.633115 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:17.635044 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":71,"threads_started":1}
W20260926 05:42:17.635177 1108 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:17.630501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:52129 (local address 127.23.213.254:38421)
0926 05:42:17.630568 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:17.630570 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:17.630602 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:42:17.630690 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:17.630691 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:17.630703 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:17.630749 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:17.631073 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.631570 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.632277 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:17.632471 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:17.632522 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:17.635130 (+ 2608us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:52129: BlockingRecv error: recv got EOF from 127.23.213.195:52129 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 05:42:18 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:18 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:18.366359 1111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:18.361216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:46495)
0926 05:42:18.361379 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:18.361395 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:18.361477 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:18.361664 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:18.361665 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:18.361945 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:18.361997 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:18.361999 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.362776 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.362778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:18.363383 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:18.363384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.363821 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.363822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:18.363890 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:18.364328 (+ 438us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:18.364333 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:18.366300 (+ 1967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
W20260926 05:42:18.366523 1112 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:18.361301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:46495 (local address 127.23.213.254:38421)
0926 05:42:18.361423 (+ 122us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:18.361426 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:18.361452 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:42:18.361544 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:18.361545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:18.361557 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:18.361655 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:18.362100 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.362683 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.363477 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.363713 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.363807 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:18.366425 (+ 2618us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:46495: BlockingRecv error: recv got EOF from 127.23.213.193:46495 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
Sep 26 05:42:18 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:18 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:18.506069 1113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:18.500671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:45101)
0926 05:42:18.500917 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:18.500930 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:18.501035 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:18.501220 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:18.501222 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:18.501433 (+ 211us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:18.501490 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:18.501492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.502300 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.502301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:18.502927 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:18.502929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.503352 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.503353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:18.503433 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:18.503925 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:18.503930 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:18.506009 (+ 2079us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":92,"threads_started":1}
W20260926 05:42:18.506251 1112 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:18.500773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:45101 (local address 127.23.213.254:38421)
0926 05:42:18.500887 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:18.500889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:18.501016 (+ 127us) server_negotiation.cc:407] Connection header received
0926 05:42:18.501114 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:18.501115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:18.501129 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:18.501213 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:18.501581 (+ 368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.502195 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.503023 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.503246 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.503351 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:18.506164 (+ 2813us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:45101: BlockingRecv error: recv got EOF from 127.23.213.194:45101 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
Sep 26 05:42:18 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:18 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:18.640151 1114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:18.635727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:36307)
0926 05:42:18.635889 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:18.635908 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:18.635967 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:18.636080 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:18.636081 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:18.636238 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:18.636288 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:18.636290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.636955 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.636956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:18.637520 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:18.637522 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.637849 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.637850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:18.637903 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:18.638331 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:18.638336 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:18.640093 (+ 1757us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":115,"threads_started":1}
W20260926 05:42:18.640280 1112 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:18.635792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:36307 (local address 127.23.213.254:38421)
0926 05:42:18.635872 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:18.635874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:18.635970 (+ 96us) server_negotiation.cc:407] Connection header received
0926 05:42:18.636010 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:18.636011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:18.636023 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:18.636074 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:18.636364 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.636864 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.637600 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:18.637794 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:18.637842 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:18.640198 (+ 2356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:36307: BlockingRecv error: recv got EOF from 127.23.213.195:36307 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 05:42:19 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:19 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:19.371526 1115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:19.366785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:40471)
0926 05:42:19.366906 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:19.366921 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:19.366971 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:19.367174 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:19.367175 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:19.367331 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:19.367381 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:19.367382 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.368053 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.368054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:19.368578 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:19.368580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.368953 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.368953 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:19.368995 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:19.369517 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:19.369522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:19.371455 (+ 1933us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
W20260926 05:42:19.371641 1116 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:19.366933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:40471 (local address 127.23.213.254:38421)
0926 05:42:19.367052 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:19.367056 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:19.367068 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:42:19.367090 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:19.367090 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:19.367099 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:19.367158 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:19.367449 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.367986 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.368653 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.368884 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.368940 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:19.371561 (+ 2621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:40471: BlockingRecv error: recv got EOF from 127.23.213.193:40471 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":71,"threads_started":1}
Sep 26 05:42:19 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:19 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:19.511477 1117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:19.506662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:33313)
0926 05:42:19.506836 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:19.506846 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:19.506903 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:19.507090 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:19.507091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:19.507242 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:19.507311 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:19.507313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.508088 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.508088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:19.508594 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:19.508595 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.508984 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.508985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:19.509029 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:19.509521 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:19.509526 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:19.511418 (+ 1892us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":114,"threads_started":1}
W20260926 05:42:19.511595 1116 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:19.506711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:33313 (local address 127.23.213.254:38421)
0926 05:42:19.506786 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:19.506788 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:19.506977 (+ 189us) server_negotiation.cc:407] Connection header received
0926 05:42:19.507005 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:19.507006 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:19.507017 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:19.507076 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:19.507382 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.508021 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.508672 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.508917 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.508972 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:19.511519 (+ 2547us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:33313: BlockingRecv error: recv got EOF from 127.23.213.194:33313 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 05:42:19 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:19 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:19.645291 1118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:19.640673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:49943)
0926 05:42:19.640866 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:19.640877 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:19.640940 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:19.641115 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:19.641116 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:19.641293 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:19.641346 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:19.641348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.641998 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.641999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:19.642536 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:19.642537 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.642857 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.642857 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:19.642899 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:19.643375 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:19.643379 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:19.645187 (+ 1808us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":78,"threads_started":1}
W20260926 05:42:19.645363 1116 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:19.640785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:49943 (local address 127.23.213.254:38421)
0926 05:42:19.640849 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:19.640851 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:19.640936 (+ 85us) server_negotiation.cc:407] Connection header received
0926 05:42:19.640982 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:19.640983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:19.641006 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:19.641053 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:19.641417 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.641932 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.642613 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:19.642788 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:19.642842 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:19.645286 (+ 2444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:49943: BlockingRecv error: recv got EOF from 127.23.213.195:49943 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 05:42:20 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:20 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:20.377377 1119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:20.372084 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:55535)
0926 05:42:20.372232 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:20.372243 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:20.372301 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:20.372562 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:20.372563 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:20.372723 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:20.372776 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:20.372777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.373563 (+ 786us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.373565 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:20.374312 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:20.374316 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.374671 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.374672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:20.374753 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:20.375268 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:20.375287 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:20.377304 (+ 2017us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
W20260926 05:42:20.377492 1120 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:20.372233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:55535 (local address 127.23.213.254:38421)
0926 05:42:20.372396 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:20.372398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:20.372407 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:20.372432 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:20.372432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:20.372443 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:20.372520 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:20.372882 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.373447 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.374400 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.374599 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.374659 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:20.377415 (+ 2756us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:55535: BlockingRecv error: recv got EOF from 127.23.213.193:55535 (error 108)
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":103,"threads_started":1}
Sep 26 05:42:20 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:20 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:20.519951 1121 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:20.512102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:38297)
0926 05:42:20.512293 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:20.512305 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:20.512380 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:20.512564 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:20.512566 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:20.512748 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:20.512796 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:20.512797 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.513461 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.513462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:20.514085 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:20.514087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.514467 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.514468 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:20.514541 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:20.518020 (+ 3479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:20.518025 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:20.519883 (+ 1858us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":102,"threads_started":1}
W20260926 05:42:20.520090 1120 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:20.512216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:38297 (local address 127.23.213.254:38421)
0926 05:42:20.512289 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:20.512291 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:20.512360 (+ 69us) server_negotiation.cc:407] Connection header received
0926 05:42:20.512456 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:20.512457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:20.512469 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:20.512557 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:20.512878 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.513382 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.514170 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.514367 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.514455 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:20.519977 (+ 5522us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:38297: BlockingRecv error: recv got EOF from 127.23.213.194:38297 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 05:42:20 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:20 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:20.650859 1122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:20.645870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:55715)
0926 05:42:20.646043 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:20.646055 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:20.646126 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:20.646266 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:20.646266 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:20.646445 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:20.646496 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:20.646498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.647518 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.647519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:20.648103 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:20.648105 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.648472 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.648473 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:20.648532 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:20.648961 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:20.648965 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:20.650795 (+ 1830us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":114,"threads_started":1}
W20260926 05:42:20.651008 1120 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:20.645925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:55715 (local address 127.23.213.254:38421)
0926 05:42:20.645969 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:20.645971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:20.646131 (+ 160us) server_negotiation.cc:407] Connection header received
0926 05:42:20.646174 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:20.646174 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:20.646186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:20.646244 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:20.646567 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.647299 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.648202 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:20.648396 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:20.648477 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:20.650929 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:55715: BlockingRecv error: recv got EOF from 127.23.213.195:55715 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 05:42:21 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:21 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:21.382805 1123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:21.377983 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:33555)
0926 05:42:21.378137 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:21.378148 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:21.378198 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:21.378341 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:21.378342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:21.378545 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:21.378596 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:21.378598 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.379309 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.379310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:21.379874 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:21.379877 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.380208 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.380208 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:21.380248 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:21.380682 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:21.380686 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:21.382748 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
W20260926 05:42:21.382947 1124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:21.378056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:33555 (local address 127.23.213.254:38421)
0926 05:42:21.378185 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:21.378186 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:21.378195 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:21.378260 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:21.378260 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:21.378270 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:21.378320 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:21.378659 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.379240 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.379945 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.380144 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.380196 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:21.382843 (+ 2647us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:33555: BlockingRecv error: recv got EOF from 127.23.213.193:33555 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
Sep 26 05:42:21 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:21 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:21.525393 1125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:21.520445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:39487)
0926 05:42:21.520632 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:21.520644 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:21.520725 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:21.520902 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:21.520904 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:21.521135 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:21.521202 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:21.521206 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.521953 (+ 747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.521954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:21.522764 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:21.522766 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.523087 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.523088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:21.523126 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:21.523518 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:21.523533 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:21.525297 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":63,"threads_started":1}
W20260926 05:42:21.525440 1124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:21.520510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:39487 (local address 127.23.213.254:38421)
0926 05:42:21.520579 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:21.520583 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:21.520727 (+ 144us) server_negotiation.cc:407] Connection header received
0926 05:42:21.520782 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:21.520783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:21.520796 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:21.520884 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:21.521281 (+ 397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.521876 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.522831 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.523026 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.523078 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:21.525376 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:39487: BlockingRecv error: recv got EOF from 127.23.213.194:39487 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 05:42:21 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:21 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:21.656239 1126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:21.651326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:54405)
0926 05:42:21.651582 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:21.651593 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:21.651646 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:21.651830 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:21.651831 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:21.652054 (+ 223us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:21.652112 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:21.652116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.652880 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.652881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:21.653415 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:21.653417 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.653884 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.653885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:21.653952 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:21.654364 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:21.654368 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:21.656145 (+ 1777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":54,"threads_started":1}
W20260926 05:42:21.656278 1124 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:21.651611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:54405 (local address 127.23.213.254:38421)
0926 05:42:21.651669 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:21.651671 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:21.651701 (+ 30us) server_negotiation.cc:407] Connection header received
0926 05:42:21.651731 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:21.651732 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:21.651758 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:21.651797 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:21.652199 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.652813 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.653509 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:21.653687 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:21.653740 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:21.656217 (+ 2477us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:54405: BlockingRecv error: recv got EOF from 127.23.213.195:54405 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 26 05:42:22 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:22 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:22.388002 1127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:22.383349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:46795)
0926 05:42:22.383511 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:22.383538 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:22.383603 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:22.383750 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:22.383751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:22.383931 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:22.383983 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:22.383985 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.384650 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.384651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:22.385195 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:22.385196 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.385526 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.385527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:22.385570 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:22.386037 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:22.386042 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:22.387895 (+ 1853us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
W20260926 05:42:22.388087 1128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:22.383444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:46795 (local address 127.23.213.254:38421)
0926 05:42:22.383592 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:22.383594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:22.383603 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:22.383654 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:22.383654 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:22.383664 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:22.383725 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:22.384054 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.384554 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.385269 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.385459 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.385508 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:22.387983 (+ 2475us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:46795: BlockingRecv error: recv got EOF from 127.23.213.193:46795 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":81,"threads_started":1}
Sep 26 05:42:22 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:22 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:22.531275 1129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:22.526171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:53299)
0926 05:42:22.526333 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:22.526352 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:22.526422 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:22.526604 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:22.526605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:22.526806 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:22.526862 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:22.526864 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.527521 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.527522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:22.528351 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:22.528355 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.528738 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.528739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:22.528800 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:22.529300 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:22.529306 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:22.531168 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":109,"threads_started":1}
W20260926 05:42:22.531518 1128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:22.526239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:53299 (local address 127.23.213.254:38421)
0926 05:42:22.526319 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:22.526321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:22.526422 (+ 101us) server_negotiation.cc:407] Connection header received
0926 05:42:22.526506 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:22.526507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:22.526518 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:22.526578 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:22.526932 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.527424 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.528433 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.528627 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.528682 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:22.531432 (+ 2750us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:53299: BlockingRecv error: recv got EOF from 127.23.213.194:53299 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 05:42:22 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:22 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:22.661433 1130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:22.656763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:40477)
0926 05:42:22.656943 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:22.656953 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:22.657008 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:22.657122 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:22.657123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:22.657287 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:22.657335 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:22.657336 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.658043 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.658043 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:22.658620 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:22.658622 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.658957 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.658958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:22.659012 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:22.659559 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:22.659563 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:22.661321 (+ 1758us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":87,"threads_started":1}
W20260926 05:42:22.661463 1128 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:22.656810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:40477 (local address 127.23.213.254:38421)
0926 05:42:22.656872 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:22.656874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:22.657006 (+ 132us) server_negotiation.cc:407] Connection header received
0926 05:42:22.657055 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:22.657056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:22.657068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:22.657117 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:22.657394 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.657942 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.658694 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:22.658878 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:22.658926 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:22.661396 (+ 2470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:40477: BlockingRecv error: recv got EOF from 127.23.213.195:40477 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 05:42:23 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:23 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:23.393395 1131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:23.388441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:33153)
0926 05:42:23.388636 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:23.388650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:23.388725 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:23.388894 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:23.388895 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:23.389060 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:23.389108 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:23.389110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.389774 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.389775 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:23.390359 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:23.390360 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.390708 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.390708 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:23.390751 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:23.391185 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:23.391189 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:23.393328 (+ 2139us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":141,"threads_started":1}
W20260926 05:42:23.393535 1132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:23.388526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:33153 (local address 127.23.213.254:38421)
0926 05:42:23.388656 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:23.388659 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:23.388716 (+ 57us) server_negotiation.cc:407] Connection header received
0926 05:42:23.388770 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:23.388771 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:23.388813 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:23.388878 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:23.389173 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.389687 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.390435 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.390646 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.390697 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:23.393437 (+ 2740us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:33153: BlockingRecv error: recv got EOF from 127.23.213.193:33153 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":86,"threads_started":1}
Sep 26 05:42:23 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:23 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:23.536623 1133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:23.531799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:35993)
0926 05:42:23.532001 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:23.532012 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:23.532078 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:23.532194 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:23.532194 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:23.532357 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:23.532403 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:23.532404 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.533090 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.533091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:23.533617 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:23.533618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.534084 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.534085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:23.534153 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:23.534673 (+ 520us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:23.534678 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:23.536517 (+ 1839us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":146,"threads_started":1}
W20260926 05:42:23.536667 1132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:23.531885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:35993 (local address 127.23.213.254:38421)
0926 05:42:23.531960 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:23.531962 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:23.532079 (+ 117us) server_negotiation.cc:407] Connection header received
0926 05:42:23.532126 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:23.532127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:23.532138 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:23.532188 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:23.532498 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.533016 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.533718 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.533984 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.534070 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:23.536602 (+ 2532us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:35993: BlockingRecv error: recv got EOF from 127.23.213.194:35993 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 05:42:23 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:23 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:23.667174 1134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:23.662030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:48871)
0926 05:42:23.662283 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:23.662295 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:23.662447 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:23.662714 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:23.662715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:23.662928 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:23.662974 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:23.662976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.663681 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.663683 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:23.664473 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:23.664476 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.664810 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.664811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:23.664850 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:23.665283 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:23.665286 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:23.667118 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":95,"threads_started":1}
W20260926 05:42:23.667352 1132 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:23.662138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:48871 (local address 127.23.213.254:38421)
0926 05:42:23.662274 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:23.662276 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:23.662418 (+ 142us) server_negotiation.cc:407] Connection header received
0926 05:42:23.662602 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:23.662603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:23.662619 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:23.662705 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:23.663057 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.663577 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.664548 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:23.664743 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:23.664796 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:23.667230 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:48871: BlockingRecv error: recv got EOF from 127.23.213.195:48871 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
Sep 26 05:42:24 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:24 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:24.398566 1135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:24.393947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:45589)
0926 05:42:24.394086 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:24.394105 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:24.394153 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:24.394315 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:24.394316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:24.394481 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:24.394528 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:24.394530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.395191 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.395193 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:24.395794 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:24.395796 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.396158 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.396159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:24.396209 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:24.396631 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:24.396636 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:24.398462 (+ 1826us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":85,"threads_started":1}
W20260926 05:42:24.398599 1136 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:24.393995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:45589 (local address 127.23.213.254:38421)
0926 05:42:24.394127 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:24.394134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:24.394142 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:24.394210 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:24.394210 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:24.394220 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:24.394278 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:24.394588 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.395084 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.395879 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.396078 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.396127 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:24.398534 (+ 2407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:45589: BlockingRecv error: recv got EOF from 127.23.213.193:45589 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":77,"threads_started":1}
Sep 26 05:42:24 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:24 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:24.541733 1137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:24.537102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:59335)
0926 05:42:24.537234 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:24.537244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:24.537323 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:24.537474 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:24.537475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:24.537660 (+ 185us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:24.537704 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:24.537706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.538413 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.538414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:24.538958 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:24.538959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.539343 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.539344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:24.539404 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:24.539875 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:24.539879 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:24.541667 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":91,"threads_started":1}
W20260926 05:42:24.541920 1136 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:24.537236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:59335 (local address 127.23.213.254:38421)
0926 05:42:24.537314 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:24.537316 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:24.537324 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:24.537383 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:24.537384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:24.537396 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:24.537468 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:24.537851 (+ 383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.538326 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.539057 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.539256 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.539342 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:24.541839 (+ 2497us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:59335: BlockingRecv error: recv got EOF from 127.23.213.194:59335 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 26 05:42:24 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:24 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:24.672570 1138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:24.667679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:57661)
0926 05:42:24.667921 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:24.667934 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:24.668006 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:24.668142 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:24.668142 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:24.668331 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:24.668383 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:24.668384 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.669009 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.669010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:24.669561 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:24.669563 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.669954 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.669956 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:24.670002 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:24.670560 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:24.670565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:24.672497 (+ 1932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":223,"thread_start_us":189,"threads_started":1}
W20260926 05:42:24.672694 1136 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:24.667772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:57661 (local address 127.23.213.254:38421)
0926 05:42:24.667836 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:24.667838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:24.668005 (+ 167us) server_negotiation.cc:407] Connection header received
0926 05:42:24.668053 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:24.668054 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:24.668065 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:24.668116 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:24.668448 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.668931 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.669639 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:24.669871 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:24.669921 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:24.672592 (+ 2671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:57661: BlockingRecv error: recv got EOF from 127.23.213.195:57661 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 26 05:42:25 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:25 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:25.403925 1139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:25.399055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:58671)
0926 05:42:25.399255 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:25.399268 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:25.399344 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:25.399507 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:25.399508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:25.399686 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:25.399748 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:25.399750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.400399 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.400400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:25.401012 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:25.401014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.401348 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.401349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:25.401396 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:25.401954 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:25.401958 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:25.403863 (+ 1905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":130,"threads_started":1}
W20260926 05:42:25.404055 1140 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:25.399127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:58671 (local address 127.23.213.254:38421)
0926 05:42:25.399260 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:25.399262 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:25.399328 (+ 66us) server_negotiation.cc:407] Connection header received
0926 05:42:25.399390 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:25.399390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:25.399400 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:25.399459 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:25.399823 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.400321 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.401089 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.401277 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.401327 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:25.403962 (+ 2635us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:58671: BlockingRecv error: recv got EOF from 127.23.213.193:58671 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":78,"threads_started":1}
Sep 26 05:42:25 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:25 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:25.550391 1141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:25.542288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:39465)
0926 05:42:25.542464 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:25.542479 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:25.542567 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:25.542741 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:25.542742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:25.542937 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:25.542987 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:25.542989 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.543636 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.543637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:25.544246 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:25.544247 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.544570 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.544571 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:25.544617 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:25.548182 (+ 3565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:25.548190 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:25.550289 (+ 2099us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":71,"threads_started":1}
W20260926 05:42:25.550451 1140 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:25.542410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:39465 (local address 127.23.213.254:38421)
0926 05:42:25.542479 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:25.542481 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:25.542541 (+ 60us) server_negotiation.cc:407] Connection header received
0926 05:42:25.542634 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:25.542634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:25.542645 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:25.542685 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:25.543065 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.543558 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.544319 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.544503 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.544551 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:25.550384 (+ 5833us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:39465: BlockingRecv error: recv got EOF from 127.23.213.194:39465 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 05:42:25 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:25 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.195@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:25.678974 1142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:25.673162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.195:54189)
0926 05:42:25.673325 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:25.673339 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:25.673421 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:25.673577 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:25.673578 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:25.673845 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:25.673907 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:25.673909 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.674644 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.674646 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:25.675587 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:25.675590 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.675979 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.675980 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:25.676047 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:25.676537 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:25.676543 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:25.678885 (+ 2342us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":85,"threads_started":1}
W20260926 05:42:25.679090 1140 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:25.673271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:54189 (local address 127.23.213.254:38421)
0926 05:42:25.673329 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:25.673332 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:25.673428 (+ 96us) server_negotiation.cc:407] Connection header received
0926 05:42:25.673488 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:25.673489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:25.673502 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:25.673551 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:25.674013 (+ 462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.674560 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.675687 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:25.675893 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:25.675953 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:25.679009 (+ 3056us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.195:54189: BlockingRecv error: recv got EOF from 127.23.213.195:54189 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 26 05:42:26 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:26 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.193@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:26.409171 1143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:26.404573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.193:40949)
0926 05:42:26.404756 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:26.404768 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:26.404828 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:26.404997 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:26.404997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:26.405252 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:26.405312 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:26.405315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.406036 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.406037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:26.406586 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:26.406587 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.406906 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.406906 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:26.406944 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:26.407426 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:26.407433 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:26.409103 (+ 1670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
W20260926 05:42:26.409240 1144 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:26.404683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:40949 (local address 127.23.213.254:38421)
0926 05:42:26.404810 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:26.404813 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:26.404825 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:42:26.404883 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:26.404884 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:26.404896 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:26.404962 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:26.405398 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.405972 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.406663 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.406843 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.406891 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:26.409189 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.193:40949: BlockingRecv error: recv got EOF from 127.23.213.193:40949 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":51,"threads_started":1}
Sep 26 05:42:26 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
Sep 26 05:42:26 dist-test-slave-1fdt krb5kdc[32554](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.23.213.194@KRBTEST.COM for oryx/127.23.213.254@KRBTEST.COM, Server not found in Kerberos database
I20260926 05:42:26.555889 1145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:26.551082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38421 (local address 127.23.213.194:38667)
0926 05:42:26.551267 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:26.551280 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:26.551392 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:26.551540 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:26.551541 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:26.551737 (+ 196us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:26.551784 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:26.551787 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.552491 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.552492 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:26.553094 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:26.553096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.553475 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.553476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:26.553537 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:26.554014 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:26.554025 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:26.555823 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.23.213.254:38421: Server oryx/127.23.213.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":94,"threads_started":1}
W20260926 05:42:26.556052 1144 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:26.551172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:38667 (local address 127.23.213.254:38421)
0926 05:42:26.551268 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:26.551271 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:26.551364 (+ 93us) server_negotiation.cc:407] Connection header received
0926 05:42:26.551471 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:26.551472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:26.551484 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:26.551536 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:26.551869 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.552404 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.553178 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.553396 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.553476 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:26.555974 (+ 2498us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.23.213.194:38667: BlockingRecv error: recv got EOF from 127.23.213.194:38667 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 05:42:26.600188 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 669
I20260926 05:42:26.604755 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 821
I20260926 05:42:26.609194 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 953
I20260926 05:42:26.613248 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 599
2026-09-26T05:42:26Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17459 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T05:42:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:42:26Z Disabled control of system clock
I20260926 05:42:26.641103 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:44999
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:44999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 05:42:26.721877 1149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:26.722084 1149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:26.722105 1149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:26.723249 1149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:26.723285 1149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:26.723299 1149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:26.723310 1149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:26.723322 1149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:26.724640 1149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:44999
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:44999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:26.724915 1149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:26.725174 1149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:26.730110 1155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:26.730170 1154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:26.730293 1149 server_base.cc:1034] running on GCE node
W20260926 05:42:26.730110 1157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:26.730618 1149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:26.730933 1149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:26.732160 1149 hybrid_clock.cc:648] HybridClock initialized: now 1790401346732138 us; error 34 us; skew 500 ppm
I20260926 05:42:26.733553 1149 webserver.cc:460] Webserver started at http://127.23.213.254:42373/ using document root <none> and password file <none>
I20260926 05:42:26.733906 1149 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:26.733963 1149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:26.734145 1149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:26.735342 1149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "d06d83ca2cc549bda217f13134377dfa"
format_stamp: "Formatted at 2026-09-26 05:42:26 on dist-test-slave-1fdt"
server_key: "70f2e0fb2edaa6f5be4a9b39945bc8c6"
server_key_iv: "08c9fb68c83efc51a89990cd71203abc"
server_key_version: "encryptionkey@0"
I20260926 05:42:26.735744 1149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "d06d83ca2cc549bda217f13134377dfa"
format_stamp: "Formatted at 2026-09-26 05:42:26 on dist-test-slave-1fdt"
server_key: "70f2e0fb2edaa6f5be4a9b39945bc8c6"
server_key_iv: "08c9fb68c83efc51a89990cd71203abc"
server_key_version: "encryptionkey@0"
I20260926 05:42:26.737162 1149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:42:26.738006 1163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:26.738204 1149 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:42:26.738322 1149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "d06d83ca2cc549bda217f13134377dfa"
format_stamp: "Formatted at 2026-09-26 05:42:26 on dist-test-slave-1fdt"
server_key: "70f2e0fb2edaa6f5be4a9b39945bc8c6"
server_key_iv: "08c9fb68c83efc51a89990cd71203abc"
server_key_version: "encryptionkey@0"
I20260926 05:42:26.738405 1149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:26.776316 1149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:26.776702 1149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:26.776873 1149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:26.781162 1149 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:44999
I20260926 05:42:26.781198 1215 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:44999 every 8 connection(s)
I20260926 05:42:26.781574 1149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:42:26.782275 1216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:26.785151 1216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa: Bootstrap starting.
I20260926 05:42:26.785815 1216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:26.785826 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1149
I20260926 05:42:26.785945 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:42:26.786173 24407 external_mini_cluster.cc:1428] Setting key 5ad8cad104f08cdf9460b113be71e2ec
I20260926 05:42:26.786164 1216 log.cc:824] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:26.787199 1216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa: No bootstrap required, opened a new log
I20260926 05:42:26.789332 1216 raft_consensus.cc:348] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06d83ca2cc549bda217f13134377dfa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44999 } }
I20260926 05:42:26.789518 1216 raft_consensus.cc:374] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:26.789559 1216 raft_consensus.cc:729] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d06d83ca2cc549bda217f13134377dfa, State: Initialized, Role: FOLLOWER
I20260926 05:42:26.789675 1216 consensus_queue.cc:260] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06d83ca2cc549bda217f13134377dfa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44999 } }
I20260926 05:42:26.789795 1216 raft_consensus.cc:388] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:42:26.789852 1216 raft_consensus.cc:482] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:42:26.789903 1216 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:26.790596 1216 raft_consensus.cc:504] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06d83ca2cc549bda217f13134377dfa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44999 } }
I20260926 05:42:26.790711 1216 leader_election.cc:302] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d06d83ca2cc549bda217f13134377dfa; no voters:
I20260926 05:42:26.790864 1216 leader_election.cc:288] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:42:26.790916 1221 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:26.791182 1216 sys_catalog.cc:564] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:26.791402 1221 raft_consensus.cc:686] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [term 1 LEADER]: Becoming Leader. State: Replica: d06d83ca2cc549bda217f13134377dfa, State: Running, Role: LEADER
I20260926 05:42:26.791585 1221 consensus_queue.cc:237] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06d83ca2cc549bda217f13134377dfa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44999 } }
I20260926 05:42:26.792198 1222 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d06d83ca2cc549bda217f13134377dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06d83ca2cc549bda217f13134377dfa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44999 } } }
I20260926 05:42:26.792296 1222 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:26.792341 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:26.786775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50238 (local address 127.23.213.254:44999)
0926 05:42:26.786977 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:26.786986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:26.787009 (+ 23us) server_negotiation.cc:407] Connection header received
0926 05:42:26.787493 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:26.787501 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:26.787569 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:26.787695 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:26.787934 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.788867 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.790020 (+ 1153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.790635 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.790769 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:26.791535 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:26.791540 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:26.791540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:26.791757 (+ 217us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:26.791800 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:26.791816 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:26.791860 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:26.792024 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 05:42:26.792107 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":95,"threads_started":1}
I20260926 05:42:26.792687 1223 sys_catalog.cc:455] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [sys.catalog]: SysCatalogTable state changed. Reason: New leader d06d83ca2cc549bda217f13134377dfa. Latest consensus state: current_term: 1 leader_uuid: "d06d83ca2cc549bda217f13134377dfa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d06d83ca2cc549bda217f13134377dfa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44999 } } }
I20260926 05:42:26.792935 1223 sys_catalog.cc:458] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa [sys.catalog]: This master's current role is: LEADER
W20260926 05:42:26.793154 1237 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:42:26.793217 1237 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:42:26.793280 1232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:42:26.793444 1232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:42:26.794994 1232 catalog_manager.cc:1347] Generated new cluster ID: 4a29dab1fad6435c8bf4f942f82bdae7
I20260926 05:42:26.795044 1232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:42:26.803432 1232 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:42:26.804028 1232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:42:26.813194 1232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d06d83ca2cc549bda217f13134377dfa: Generated new TSK 0
I20260926 05:42:26.813409 1232 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:42:26.814666 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44999
--builtin_ntp_servers=127.23.213.212:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 05:42:26.897147 1240 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:26.897363 1240 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:26.897387 1240 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:26.898633 1240 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:26.898674 1240 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:26.898699 1240 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:42:26.900105 1240 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1240
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:26.900365 1240 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:26.900668 1240 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:26.906553 1245 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:26.906561 1246 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:26.906566 1248 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:26.906826 1240 server_base.cc:1034] running on GCE node
I20260926 05:42:26.907029 1240 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:26.907274 1240 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:26.908445 1240 hybrid_clock.cc:648] HybridClock initialized: now 1790401346908430 us; error 31 us; skew 500 ppm
I20260926 05:42:26.909734 1240 webserver.cc:460] Webserver started at http://127.23.213.193:40073/ using document root <none> and password file <none>
I20260926 05:42:26.910074 1240 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:26.910144 1240 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:26.910285 1240 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:26.911355 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "678aaf9310be4e1aa29f79fc671d86fb"
format_stamp: "Formatted at 2026-09-26 05:42:26 on dist-test-slave-1fdt"
server_key: "f04ab5adffdbe02ed29c62238c6c4304"
server_key_iv: "0be382ec8cb07a78046bffe7ef28a00c"
server_key_version: "encryptionkey@0"
I20260926 05:42:26.911722 1240 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "678aaf9310be4e1aa29f79fc671d86fb"
format_stamp: "Formatted at 2026-09-26 05:42:26 on dist-test-slave-1fdt"
server_key: "f04ab5adffdbe02ed29c62238c6c4304"
server_key_iv: "0be382ec8cb07a78046bffe7ef28a00c"
server_key_version: "encryptionkey@0"
I20260926 05:42:26.913451 1240 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260926 05:42:26.914400 1254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:26.914580 1240 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:42:26.914674 1240 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "678aaf9310be4e1aa29f79fc671d86fb"
format_stamp: "Formatted at 2026-09-26 05:42:26 on dist-test-slave-1fdt"
server_key: "f04ab5adffdbe02ed29c62238c6c4304"
server_key_iv: "0be382ec8cb07a78046bffe7ef28a00c"
server_key_version: "encryptionkey@0"
I20260926 05:42:26.914768 1240 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:26.922291 1240 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:26.922602 1240 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:26.922752 1240 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:26.922968 1240 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:26.923297 1240 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:26.923352 1240 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:26.923393 1240 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:26.923421 1240 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:26.929458 1240 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:44763
I20260926 05:42:26.929530 1367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:44763 every 8 connection(s)
I20260926 05:42:26.929939 1240 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:42:26.935019 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:26.930686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:49893 (local address 127.23.213.254:44999)
0926 05:42:26.930726 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:26.930729 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:26.930993 (+ 264us) server_negotiation.cc:407] Connection header received
0926 05:42:26.931499 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:26.931500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:26.931508 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:26.931566 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:26.932191 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.932716 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.933658 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.933935 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.934001 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:26.934787 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:26.934788 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:26.934788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:26.934792 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:26.934805 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:26.934808 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:26.934849 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:26.934928 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 05:42:26.934949 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:42:26.935266 1370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:26.930662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:44999 (local address 127.23.213.193:49893)
0926 05:42:26.930862 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:26.930905 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:26.931415 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:26.931631 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:26.931634 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:26.931643 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:26.932124 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:26.932128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.932803 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.932804 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:26.933588 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:26.933589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:26.934041 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:26.934048 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:26.934171 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:26.934659 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:42:26.934660 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:26.934688 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:42:26.934709 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:26.934849 (+ 140us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:26.934852 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:26.934854 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:42:26.934929 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:42:26.935006 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260926 05:42:26.935860 1368 heartbeater.cc:347] Connected to a master server at 127.23.213.254:44999
I20260926 05:42:26.935981 1368 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:26.936159 1368 heartbeater.cc:511] Master 127.23.213.254:44999 requested a full tablet report, sending...
I20260926 05:42:26.936681 1180 ts_manager.cc:194] Registered new tserver with Master: 678aaf9310be4e1aa29f79fc671d86fb (127.23.213.193:44763)
I20260926 05:42:26.937242 1180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.193:49893
I20260926 05:42:26.939642 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1240
I20260926 05:42:26.939713 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:42:26.939860 24407 external_mini_cluster.cc:1428] Setting key da609f87d5f1ca04f8b64809a646692e
I20260926 05:42:26.940654 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44999
--builtin_ntp_servers=127.23.213.212:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 05:42:27.024355 1371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:27.024616 1371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:27.024657 1371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:27.025996 1371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:27.026067 1371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:27.026113 1371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:42:27.027544 1371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1371
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:27.027806 1371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:27.028079 1371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:27.033273 1379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.033394 1377 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.033371 1376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:27.033582 1371 server_base.cc:1034] running on GCE node
I20260926 05:42:27.033864 1371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:27.034066 1371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:27.035216 1371 hybrid_clock.cc:648] HybridClock initialized: now 1790401347035190 us; error 40 us; skew 500 ppm
I20260926 05:42:27.036646 1371 webserver.cc:460] Webserver started at http://127.23.213.194:45913/ using document root <none> and password file <none>
I20260926 05:42:27.036913 1371 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:27.036965 1371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:27.037148 1371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:27.038427 1371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "8cec32e4153a4ca48614012869ccb2c4"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "e2b9669a7e95f167165c2ea73b093457"
server_key_iv: "1e878dac5ee43f4b60cdcac7ab38c290"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.038825 1371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "8cec32e4153a4ca48614012869ccb2c4"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "e2b9669a7e95f167165c2ea73b093457"
server_key_iv: "1e878dac5ee43f4b60cdcac7ab38c290"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.040342 1371 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:42:27.041146 1385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.041363 1371 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:27.041462 1371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "8cec32e4153a4ca48614012869ccb2c4"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "e2b9669a7e95f167165c2ea73b093457"
server_key_iv: "1e878dac5ee43f4b60cdcac7ab38c290"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.041544 1371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:27.061887 1371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:27.062227 1371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:27.062390 1371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:27.062657 1371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:27.063033 1371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:27.063071 1371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.063118 1371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:27.063151 1371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.069569 1371 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:44213
I20260926 05:42:27.069607 1498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:44213 every 8 connection(s)
I20260926 05:42:27.070045 1371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:42:27.075325 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.070844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:36561 (local address 127.23.213.254:44999)
0926 05:42:27.070907 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.070909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.071153 (+ 244us) server_negotiation.cc:407] Connection header received
0926 05:42:27.071646 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.071647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.071655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.071713 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.072399 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.072928 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.074065 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.074256 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.074313 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:27.075093 (+ 780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:27.075094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:27.075094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:27.075098 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:27.075112 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:27.075116 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:27.075158 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:27.075247 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 05:42:27.075268 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:42:27.075565 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1371
I20260926 05:42:27.075575 1501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.070823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:44999 (local address 127.23.213.194:36561)
0926 05:42:27.071019 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:27.071061 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:27.071566 (+ 505us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:27.071777 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:27.071780 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:27.071788 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:27.072291 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:27.072296 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.073033 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.073035 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:27.073983 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:27.073987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.074343 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.074344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:27.074459 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:27.074957 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:42:27.074958 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:27.074986 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:42:27.075008 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:27.075172 (+ 164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:27.075177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:27.075179 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:42:27.075235 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:42:27.075318 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":98,"threads_started":1}
I20260926 05:42:27.075668 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:42:27.075870 24407 external_mini_cluster.cc:1428] Setting key c8934cb054bfdb4d3c76048d11231e7d
I20260926 05:42:27.076141 1499 heartbeater.cc:347] Connected to a master server at 127.23.213.254:44999
I20260926 05:42:27.076252 1499 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:27.076438 1499 heartbeater.cc:511] Master 127.23.213.254:44999 requested a full tablet report, sending...
I20260926 05:42:27.076864 1180 ts_manager.cc:194] Registered new tserver with Master: 8cec32e4153a4ca48614012869ccb2c4 (127.23.213.194:44213)
I20260926 05:42:27.076999 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44999
--builtin_ntp_servers=127.23.213.212:43411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260926 05:42:27.077198 1180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.194:36561
W20260926 05:42:27.158587 1502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:27.158823 1502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:27.158846 1502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:27.159955 1502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:27.159986 1502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:27.160009 1502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:42:27.161253 1502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1502
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:27.161435 1502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:27.161686 1502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:27.166782 1508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.166783 1507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.166846 1510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:27.167011 1502 server_base.cc:1034] running on GCE node
I20260926 05:42:27.167232 1502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:27.167498 1502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:27.168708 1502 hybrid_clock.cc:648] HybridClock initialized: now 1790401347168696 us; error 32 us; skew 500 ppm
I20260926 05:42:27.170101 1502 webserver.cc:460] Webserver started at http://127.23.213.195:40923/ using document root <none> and password file <none>
I20260926 05:42:27.170365 1502 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:27.170416 1502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:27.170559 1502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:27.171631 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "8cbe9436fd0640c1a0b3e1cb181a5088"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "c52ebb5e5a98e9ce2c3d470232e982e0"
server_key_iv: "535f0ffa4ea50a4d968ca0f582397576"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.171983 1502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "8cbe9436fd0640c1a0b3e1cb181a5088"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "c52ebb5e5a98e9ce2c3d470232e982e0"
server_key_iv: "535f0ffa4ea50a4d968ca0f582397576"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.173256 1502 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:42:27.174096 1516 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.174275 1502 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:27.174363 1502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "8cbe9436fd0640c1a0b3e1cb181a5088"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "c52ebb5e5a98e9ce2c3d470232e982e0"
server_key_iv: "535f0ffa4ea50a4d968ca0f582397576"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.174435 1502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:27.186175 1502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:27.186455 1502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:27.186627 1502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:27.186853 1502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:27.187168 1502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:27.187201 1502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.187222 1502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:27.187235 1502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.193168 1502 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:45389
I20260926 05:42:27.193271 1629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:45389 every 8 connection(s)
I20260926 05:42:27.193584 1502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:42:27.198944 1220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.194437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:41763 (local address 127.23.213.254:44999)
0926 05:42:27.194496 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.194498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.194790 (+ 292us) server_negotiation.cc:407] Connection header received
0926 05:42:27.195218 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.195219 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.195231 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.195302 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.195989 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.196498 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.197475 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.197668 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.197816 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:27.198711 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:27.198712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:27.198713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:27.198717 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:27.198732 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:27.198736 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:27.198795 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:27.198860 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 05:42:27.198882 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:42:27.199187 1632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.194425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:44999 (local address 127.23.213.195:41763)
0926 05:42:27.194620 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:27.194653 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:27.195124 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:27.195356 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:27.195359 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:27.195367 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:27.195870 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:27.195874 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.196588 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.196588 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:27.197380 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:27.197382 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:27.197811 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:27.197812 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:27.197944 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:27.198550 (+ 606us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:42:27.198551 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:27.198582 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:42:27.198605 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:27.198797 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:27.198799 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:27.198802 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:27.198850 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:42:27.198930 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":108,"threads_started":1}
I20260926 05:42:27.199782 1630 heartbeater.cc:347] Connected to a master server at 127.23.213.254:44999
I20260926 05:42:27.199940 1630 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:27.200137 1630 heartbeater.cc:511] Master 127.23.213.254:44999 requested a full tablet report, sending...
I20260926 05:42:27.200570 1180 ts_manager.cc:194] Registered new tserver with Master: 8cbe9436fd0640c1a0b3e1cb181a5088 (127.23.213.195:45389)
I20260926 05:42:27.200896 1180 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.195:41763
I20260926 05:42:27.201915 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1502
I20260926 05:42:27.201992 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:42:27.202210 24407 external_mini_cluster.cc:1428] Setting key ef04917470b2c3e406176d2818c3a8ca
I20260926 05:42:27.202953 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:42:27.204882 1220 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:27.204298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50248 (local address 127.23.213.254:44999)
0926 05:42:27.204380 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.204383 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.204493 (+ 110us) server_negotiation.cc:407] Connection header received
0926 05:42:27.204561 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.204562 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.204573 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.204619 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.204760 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50248: BlockingRecv error: recv got EOF from 127.0.0.1:50248 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:42:27.204877 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1240
I20260926 05:42:27.209432 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1371
I20260926 05:42:27.213668 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1502
I20260926 05:42:27.217931 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1149
2026-09-26T05:42:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (603 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T05:42:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:42:27Z Disabled control of system clock
I20260926 05:42:27.244818 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:44673
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:44673
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:42:27.324549 1641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:27.324797 1641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:27.324832 1641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:27.326006 1641 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:27.326081 1641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:27.326102 1641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:27.326122 1641 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:27.326151 1641 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:27.327536 1641 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:44673
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1641
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:27.327817 1641 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:27.328128 1641 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:27.333086 1646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.333060 1647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:27.333092 1641 server_base.cc:1034] running on GCE node
W20260926 05:42:27.333074 1649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:27.333642 1641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:27.333986 1641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:27.335160 1641 hybrid_clock.cc:648] HybridClock initialized: now 1790401347335148 us; error 52 us; skew 500 ppm
I20260926 05:42:27.336526 1641 webserver.cc:460] Webserver started at http://127.23.213.254:33257/ using document root <none> and password file <none>
I20260926 05:42:27.336790 1641 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:27.336860 1641 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:27.337013 1641 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:27.338181 1641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "3a01e615385e49389cf01fd5b3f8dec9"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "1b218c6de68f6eb663fb7140ef2198fa"
server_key_iv: "c12c92261c0d72e4dd670d7d676fcb63"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.338541 1641 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "3a01e615385e49389cf01fd5b3f8dec9"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "1b218c6de68f6eb663fb7140ef2198fa"
server_key_iv: "c12c92261c0d72e4dd670d7d676fcb63"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.339813 1641 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:42:27.340674 1655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.340916 1641 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:27.341022 1641 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "3a01e615385e49389cf01fd5b3f8dec9"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "1b218c6de68f6eb663fb7140ef2198fa"
server_key_iv: "c12c92261c0d72e4dd670d7d676fcb63"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.341107 1641 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:27.344519 1641 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:27.344785 1641 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:27.344926 1641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:27.348868 1641 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:44673
I20260926 05:42:27.348919 1707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:44673 every 8 connection(s)
I20260926 05:42:27.349267 1641 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:42:27.349479 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1641
I20260926 05:42:27.349581 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:42:27.349875 24407 external_mini_cluster.cc:1428] Setting key 310ba647cca5449c49d15b6ac50bb2d0
I20260926 05:42:27.350064 1708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:27.353001 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.350815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.23.213.254:44673)
0926 05:42:27.351032 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.351042 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.351073 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:42:27.351587 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.351596 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.351653 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.351807 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.352552 (+ 745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:27.352555 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:27.352555 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:27.352560 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:27.352605 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:27.352619 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:27.352692 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:27.352753 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 05:42:27.352815 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":86,"threads_started":1}
I20260926 05:42:27.353415 1708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9: Bootstrap starting.
I20260926 05:42:27.354067 1708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:27.354338 1708 log.cc:824] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:27.355018 1708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9: No bootstrap required, opened a new log
I20260926 05:42:27.356328 1708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a01e615385e49389cf01fd5b3f8dec9" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44673 } }
I20260926 05:42:27.356454 1708 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:27.356479 1708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a01e615385e49389cf01fd5b3f8dec9, State: Initialized, Role: FOLLOWER
I20260926 05:42:27.356621 1708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a01e615385e49389cf01fd5b3f8dec9" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44673 } }
I20260926 05:42:27.356695 1708 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:42:27.356717 1708 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:42:27.356743 1708 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:27.357264 1708 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a01e615385e49389cf01fd5b3f8dec9" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44673 } }
I20260926 05:42:27.357344 1708 leader_election.cc:302] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a01e615385e49389cf01fd5b3f8dec9; no voters:
I20260926 05:42:27.357551 1708 leader_election.cc:288] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:42:27.357858 1713 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:27.357862 1708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:27.358186 1713 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [term 1 LEADER]: Becoming Leader. State: Replica: 3a01e615385e49389cf01fd5b3f8dec9, State: Running, Role: LEADER
I20260926 05:42:27.358346 1713 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a01e615385e49389cf01fd5b3f8dec9" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44673 } }
I20260926 05:42:27.358778 1715 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a01e615385e49389cf01fd5b3f8dec9. Latest consensus state: current_term: 1 leader_uuid: "3a01e615385e49389cf01fd5b3f8dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a01e615385e49389cf01fd5b3f8dec9" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44673 } } }
I20260926 05:42:27.358875 1715 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:27.358908 1714 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a01e615385e49389cf01fd5b3f8dec9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a01e615385e49389cf01fd5b3f8dec9" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 44673 } } }
I20260926 05:42:27.359004 1714 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9 [sys.catalog]: This master's current role is: LEADER
W20260926 05:42:27.359730 1728 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:42:27.359787 1728 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:42:27.359854 1723 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:42:27.360033 1723 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:42:27.361526 1723 catalog_manager.cc:1347] Generated new cluster ID: b5768b8dc6f24c3abf122fdae9e60805
I20260926 05:42:27.361590 1723 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:42:27.375150 1723 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:42:27.375361 1723 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:42:27.380285 1723 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a01e615385e49389cf01fd5b3f8dec9: Generated new TSK 0
I20260926 05:42:27.380515 1723 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:42:27.387995 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44673
--builtin_ntp_servers=127.23.213.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:42:27.480504 1732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:27.480713 1732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:27.480737 1732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:27.481997 1732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:27.482038 1732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:27.482065 1732 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:42:27.483479 1732 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:27.483749 1732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:27.484081 1732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:27.490119 1737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.490226 1740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.490120 1738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:27.490288 1732 server_base.cc:1034] running on GCE node
I20260926 05:42:27.490824 1732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:27.491132 1732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:27.492313 1732 hybrid_clock.cc:648] HybridClock initialized: now 1790401347492301 us; error 32 us; skew 500 ppm
I20260926 05:42:27.493680 1732 webserver.cc:460] Webserver started at http://127.23.213.193:41947/ using document root <none> and password file <none>
I20260926 05:42:27.493976 1732 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:27.494050 1732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:27.494223 1732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:27.495309 1732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "5d863da1ba294642b0315049923cc6a5"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "0ac38fb15178a88dfee8bd1c0d6ecf16"
server_key_iv: "6da1905731e988803e7226b2e01e3e79"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.495673 1732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "5d863da1ba294642b0315049923cc6a5"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "0ac38fb15178a88dfee8bd1c0d6ecf16"
server_key_iv: "6da1905731e988803e7226b2e01e3e79"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.496989 1732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:42:27.497767 1746 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.497978 1732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:42:27.498070 1732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "5d863da1ba294642b0315049923cc6a5"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "0ac38fb15178a88dfee8bd1c0d6ecf16"
server_key_iv: "6da1905731e988803e7226b2e01e3e79"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.498138 1732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:27.502872 1732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:27.503149 1732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:27.503274 1732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:27.503487 1732 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:27.503837 1732 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:27.503882 1732 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.503909 1732 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:27.503932 1732 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.509944 1732 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:39815
I20260926 05:42:27.510017 1859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:39815 every 8 connection(s)
I20260926 05:42:27.510265 1732 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:42:27.513149 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.510951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:33019 (local address 127.23.213.254:44673)
0926 05:42:27.511016 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.511019 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.511343 (+ 324us) server_negotiation.cc:407] Connection header received
0926 05:42:27.511779 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.511780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.511787 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.511855 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.512953 (+ 1098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:27.512954 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:27.512954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:27.512958 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:27.512972 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:27.512976 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:27.513022 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:27.513084 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 05:42:27.513098 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:42:27.513342 1862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.510958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:44673 (local address 127.23.213.193:33019)
0926 05:42:27.511159 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:27.511217 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:27.511689 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:27.511923 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:27.511926 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:27.511941 (+ 15us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:27.512810 (+ 869us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:42:27.512811 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:27.512837 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:42:27.512865 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:27.513020 (+ 155us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:27.513023 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:27.513025 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:42:27.513076 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:42:27.513122 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":88,"threads_started":1}
I20260926 05:42:27.513631 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1732
I20260926 05:42:27.513713 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:42:27.513906 1860 heartbeater.cc:347] Connected to a master server at 127.23.213.254:44673
I20260926 05:42:27.513938 24407 external_mini_cluster.cc:1428] Setting key 20e9a59b7b5282a7d4c297362744e53c
I20260926 05:42:27.513993 1860 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:27.514169 1860 heartbeater.cc:511] Master 127.23.213.254:44673 requested a full tablet report, sending...
I20260926 05:42:27.514698 1672 ts_manager.cc:194] Registered new tserver with Master: 5d863da1ba294642b0315049923cc6a5 (127.23.213.193:39815)
I20260926 05:42:27.514932 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44673
--builtin_ntp_servers=127.23.213.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:42:27.594586 1863 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:27.594837 1863 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:27.594873 1863 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:27.596143 1863 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:27.596212 1863 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:27.596256 1863 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:42:27.597575 1863 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1863
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:27.597864 1863 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:27.598160 1863 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:27.603170 1869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.603180 1871 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.603184 1868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:27.603391 1863 server_base.cc:1034] running on GCE node
I20260926 05:42:27.603600 1863 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:27.603803 1863 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:27.604980 1863 hybrid_clock.cc:648] HybridClock initialized: now 1790401347604966 us; error 23 us; skew 500 ppm
I20260926 05:42:27.606383 1863 webserver.cc:460] Webserver started at http://127.23.213.194:37665/ using document root <none> and password file <none>
I20260926 05:42:27.606622 1863 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:27.606680 1863 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:27.606805 1863 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:27.607909 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "2a517484ed9d45e3ba9cbf0a4e3c5498"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "fd1e85d2eec783fb09cc4988a4ae0c40"
server_key_iv: "2966adaa6de733b8293f6758f3700686"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.608201 1863 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "2a517484ed9d45e3ba9cbf0a4e3c5498"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "fd1e85d2eec783fb09cc4988a4ae0c40"
server_key_iv: "2966adaa6de733b8293f6758f3700686"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.609472 1863 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:42:27.610180 1877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.610343 1863 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.610443 1863 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "2a517484ed9d45e3ba9cbf0a4e3c5498"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "fd1e85d2eec783fb09cc4988a4ae0c40"
server_key_iv: "2966adaa6de733b8293f6758f3700686"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.610524 1863 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:27.613811 1863 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:27.614073 1863 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:27.614216 1863 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:27.614424 1863 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:27.614737 1863 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:27.614791 1863 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.614823 1863 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:27.614849 1863 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.620535 1863 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:33445
I20260926 05:42:27.620627 1990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:33445 every 8 connection(s)
I20260926 05:42:27.620960 1863 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:42:27.624682 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.621713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:56343 (local address 127.23.213.254:44673)
0926 05:42:27.621915 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.621917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.622304 (+ 387us) server_negotiation.cc:407] Connection header received
0926 05:42:27.622898 (+ 594us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.622900 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.622908 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.622977 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.624426 (+ 1449us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:27.624428 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:27.624428 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:27.624433 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:27.624450 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:27.624454 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:27.624521 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:27.624611 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 05:42:27.624625 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166}
I20260926 05:42:27.624963 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.621687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:44673 (local address 127.23.213.194:56343)
0926 05:42:27.622137 (+ 450us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:27.622185 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:27.622701 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:27.623067 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:27.623070 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:27.623080 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:27.624245 (+ 1165us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:42:27.624246 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:27.624278 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:42:27.624311 (+ 33us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:27.624536 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:27.624539 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:27.624542 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:27.624606 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 05:42:27.624660 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":343,"threads_started":1}
I20260926 05:42:27.625502 1991 heartbeater.cc:347] Connected to a master server at 127.23.213.254:44673
I20260926 05:42:27.625590 1991 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:27.625783 1991 heartbeater.cc:511] Master 127.23.213.254:44673 requested a full tablet report, sending...
I20260926 05:42:27.626247 1672 ts_manager.cc:194] Registered new tserver with Master: 2a517484ed9d45e3ba9cbf0a4e3c5498 (127.23.213.194:33445)
I20260926 05:42:27.629457 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1863
I20260926 05:42:27.629575 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:42:27.629818 24407 external_mini_cluster.cc:1428] Setting key d734aff8c4eda9d123e663a28e84266a
I20260926 05:42:27.630757 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44673
--builtin_ntp_servers=127.23.213.212:42529
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 05:42:27.711400 1994 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:27.711614 1994 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:27.711637 1994 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:27.712873 1994 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:27.712913 1994 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:27.712937 1994 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:42:27.714378 1994 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:42529
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:44673
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1994
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:27.714619 1994 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:27.714905 1994 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:27.719869 2002 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.719856 1999 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:27.719847 2000 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:27.719985 1994 server_base.cc:1034] running on GCE node
I20260926 05:42:27.720299 1994 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:27.720536 1994 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:27.721673 1994 hybrid_clock.cc:648] HybridClock initialized: now 1790401347721665 us; error 29 us; skew 500 ppm
I20260926 05:42:27.722916 1994 webserver.cc:460] Webserver started at http://127.23.213.195:41681/ using document root <none> and password file <none>
I20260926 05:42:27.723155 1994 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:27.723201 1994 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:27.723371 1994 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:27.724452 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "b7ef0db051da425bb6f74a2e92a122c7"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "36592729d1dce8e5a3bc8e9b2783d287"
server_key_iv: "a65b926b5a335159c11814d23db770cd"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.724877 1994 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "b7ef0db051da425bb6f74a2e92a122c7"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "36592729d1dce8e5a3bc8e9b2783d287"
server_key_iv: "a65b926b5a335159c11814d23db770cd"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.726485 1994 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:42:27.727114 2008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.727289 1994 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.727356 1994 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "b7ef0db051da425bb6f74a2e92a122c7"
format_stamp: "Formatted at 2026-09-26 05:42:27 on dist-test-slave-1fdt"
server_key: "36592729d1dce8e5a3bc8e9b2783d287"
server_key_iv: "a65b926b5a335159c11814d23db770cd"
server_key_version: "encryptionkey@0"
I20260926 05:42:27.727439 1994 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:27.730795 1994 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:27.731038 1994 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:27.731175 1994 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:27.731393 1994 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:27.731683 1994 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:27.731714 1994 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.731765 1994 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:27.731792 1994 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:27.737461 1994 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:45017
I20260926 05:42:27.737540 2121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:45017 every 8 connection(s)
I20260926 05:42:27.737915 1994 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:42:27.740810 1711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.738590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:45971 (local address 127.23.213.254:44673)
0926 05:42:27.738635 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.738637 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.738863 (+ 226us) server_negotiation.cc:407] Connection header received
0926 05:42:27.739402 (+ 539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.739403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.739409 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.739466 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.740560 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:27.740569 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:27.740570 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:42:27.740574 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:27.740594 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:42:27.740598 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:27.740655 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:27.740727 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:42:27.740740 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:42:27.741086 2124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:27.738573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:44673 (local address 127.23.213.195:45971)
0926 05:42:27.738734 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:27.738765 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:27.739292 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:27.739537 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:27.739541 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:27.739551 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:27.740405 (+ 854us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:42:27.740406 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:27.740432 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:42:27.740458 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:27.740659 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:27.740664 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:27.740667 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:27.740739 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:42:27.740870 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":66,"threads_started":1}
I20260926 05:42:27.741612 2122 heartbeater.cc:347] Connected to a master server at 127.23.213.254:44673
I20260926 05:42:27.741686 2122 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:27.741869 2122 heartbeater.cc:511] Master 127.23.213.254:44673 requested a full tablet report, sending...
I20260926 05:42:27.742235 1672 ts_manager.cc:194] Registered new tserver with Master: b7ef0db051da425bb6f74a2e92a122c7 (127.23.213.195:45017)
I20260926 05:42:27.746215 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 1994
I20260926 05:42:27.746318 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:42:27.746510 24407 external_mini_cluster.cc:1428] Setting key 1c730d03fbf6c2cf8996a4b10da9f8ad
I20260926 05:42:27.747424 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 05:42:27.749226 1711 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:42:27.748850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.23.213.254:44673)
0926 05:42:27.748893 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:27.748895 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:27.748935 (+ 40us) server_negotiation.cc:407] Connection header received
0926 05:42:27.748992 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:27.748993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:27.748997 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:27.749039 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:27.749166 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43792: BlockingRecv error: recv got EOF from 127.0.0.1:43792 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:42:27.749312 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1732
I20260926 05:42:27.754311 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1863
I20260926 05:42:27.758654 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1994
I20260926 05:42:27.762768 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 1641
2026-09-26T05:42:27Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (544 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:42:27 dist-test-slave-1fdt krb5kdc[2132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:42:27 dist-test-slave-1fdt krb5kdc[2132](info): set up 2 sockets
Sep 26 05:42:27 dist-test-slave-1fdt krb5kdc[2132](info): commencing operation
krb5kdc: starting...
W20260926 05:42:28.818831 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:42:28 dist-test-slave-1fdt krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:42:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:42:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:42:28.938560 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:35885
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:35885
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:29.020342 2145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:29.020605 2145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:29.020642 2145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:29.021860 2145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:42:29.021937 2145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:29.021961 2145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:29.021982 2145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:42:29.022013 2145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:42:29.023526 2145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:35885
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:35885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:29.023819 2145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:29.024058 2145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:29.029070 2150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:29.029131 2151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:29.029197 2145 server_base.cc:1034] running on GCE node
W20260926 05:42:29.029070 2153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:29.029551 2145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:29.029937 2145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:29.031106 2145 hybrid_clock.cc:648] HybridClock initialized: now 1790401349031101 us; error 38 us; skew 500 ppm
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401349, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:29.033646 2145 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:42:29.034111 2145 webserver.cc:460] Webserver started at http://127.23.213.254:43783/ using document root <none> and password file <none>
I20260926 05:42:29.034307 2145 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:29.034348 2145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:29.034439 2145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:29.035496 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "abaf72dd3dc644e3a72b35c72bdbb800"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "badc4b42bda24864d4e4a6cc8d056bc3"
server_key_iv: "6c1496c32901a650f9930c8d51db1649"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.035811 2145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "abaf72dd3dc644e3a72b35c72bdbb800"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "badc4b42bda24864d4e4a6cc8d056bc3"
server_key_iv: "6c1496c32901a650f9930c8d51db1649"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.037048 2145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:42:29.037730 2160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.037931 2145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.038031 2145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "abaf72dd3dc644e3a72b35c72bdbb800"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "badc4b42bda24864d4e4a6cc8d056bc3"
server_key_iv: "6c1496c32901a650f9930c8d51db1649"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.038116 2145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:29.047984 2145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:29.050434 2145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:29.050601 2145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:29.054625 2145 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:35885
I20260926 05:42:29.054657 2212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:35885 every 8 connection(s)
I20260926 05:42:29.055027 2145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:42:29.055984 2213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:29.058323 2213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800: Bootstrap starting.
I20260926 05:42:29.058923 2213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:29.059226 2213 log.cc:824] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:29.059926 2213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800: No bootstrap required, opened a new log
I20260926 05:42:29.061576 2213 raft_consensus.cc:348] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35885 } }
I20260926 05:42:29.061717 2213 raft_consensus.cc:374] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:29.061795 2213 raft_consensus.cc:729] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abaf72dd3dc644e3a72b35c72bdbb800, State: Initialized, Role: FOLLOWER
I20260926 05:42:29.061981 2213 consensus_queue.cc:260] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35885 } }
I20260926 05:42:29.062067 2213 raft_consensus.cc:388] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:42:29.062114 2213 raft_consensus.cc:482] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:42:29.062165 2213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:29.063025 2213 raft_consensus.cc:504] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35885 } }
I20260926 05:42:29.063112 2213 leader_election.cc:302] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abaf72dd3dc644e3a72b35c72bdbb800; no voters:
I20260926 05:42:29.063303 2213 leader_election.cc:288] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:42:29.063411 2216 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:29.063567 2213 sys_catalog.cc:564] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:42:29.063580 2216 raft_consensus.cc:686] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [term 1 LEADER]: Becoming Leader. State: Replica: abaf72dd3dc644e3a72b35c72bdbb800, State: Running, Role: LEADER
I20260926 05:42:29.063714 2216 consensus_queue.cc:237] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35885 } }
I20260926 05:42:29.063910 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 2145
I20260926 05:42:29.064023 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:42:29.064250 24407 external_mini_cluster.cc:1428] Setting key 90f661689788624efece8ce6a72f41e9
I20260926 05:42:29.064728 2217 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35885 } } }
I20260926 05:42:29.064839 2217 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:29.065065 2218 sys_catalog.cc:455] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [sys.catalog]: SysCatalogTable state changed. Reason: New leader abaf72dd3dc644e3a72b35c72bdbb800. Latest consensus state: current_term: 1 leader_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abaf72dd3dc644e3a72b35c72bdbb800" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 35885 } } }
I20260926 05:42:29.065233 2218 sys_catalog.cc:458] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800 [sys.catalog]: This master's current role is: LEADER
I20260926 05:42:29.065264 2232 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:42:29.065861 2232 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:42:29.067481 2232 catalog_manager.cc:1347] Generated new cluster ID: f25e97b2a8164986970cd93577fa3a2a
I20260926 05:42:29.067533 2232 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401348, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:29.076330 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.064983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53444 (local address 127.23.213.254:35885)
0926 05:42:29.065417 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.065427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.065460 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:42:29.065590 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.065596 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.065657 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.065827 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:29.066573 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.067885 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.068955 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.069550 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.069642 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.072449 (+ 2807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:29.072453 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:29.072453 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:29.072474 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:29.074639 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.075042 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.075045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.075046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.075172 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.075372 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.075373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.075374 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.075704 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:29.075874 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.075963 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.076109 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":386,"thread_start_us":81,"threads_started":1}
I20260926 05:42:29.078872 2232 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:42:29.079375 2232 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:42:29.083904 2232 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800: Generated new TSK 0
I20260926 05:42:29.084079 2232 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:42:29.113504 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35885
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:29.195999 2239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:29.196203 2239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:29.196226 2239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:29.197368 2239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:29.197410 2239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:29.197436 2239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:42:29.198942 2239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:29.199194 2239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:29.199430 2239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:29.204526 2245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:29.204516 2244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:29.204525 2247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:29.204562 2239 server_base.cc:1034] running on GCE node
I20260926 05:42:29.204974 2239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:29.205231 2239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:29.206441 2239 hybrid_clock.cc:648] HybridClock initialized: now 1790401349206427 us; error 29 us; skew 500 ppm
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401349, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:29.209085 2239 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:42:29.209539 2239 webserver.cc:460] Webserver started at http://127.23.213.193:44493/ using document root <none> and password file <none>
I20260926 05:42:29.209721 2239 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:29.209827 2239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:29.209929 2239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:29.210968 2239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "a9356417d2b34dd0923e23759a7982c3"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "c34fe46299c98489b7e0b663bc2e60c0"
server_key_iv: "0752a4c845cfc9c46da5470ff3901d93"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.211268 2239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "a9356417d2b34dd0923e23759a7982c3"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "c34fe46299c98489b7e0b663bc2e60c0"
server_key_iv: "0752a4c845cfc9c46da5470ff3901d93"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.212567 2239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:42:29.213286 2254 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.213488 2239 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:29.213577 2239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "a9356417d2b34dd0923e23759a7982c3"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "c34fe46299c98489b7e0b663bc2e60c0"
server_key_iv: "0752a4c845cfc9c46da5470ff3901d93"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.213634 2239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:29.220505 2239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:29.222868 2239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:29.222982 2239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:29.223177 2239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:29.223428 2239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:29.223459 2239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.223480 2239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:29.223510 2239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.229518 2239 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:42689
I20260926 05:42:29.229563 2367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:42689 every 8 connection(s)
I20260926 05:42:29.229951 2239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401349, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:29.238715 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 2239
I20260926 05:42:29.238766 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.230706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:34303 (local address 127.23.213.254:35885)
0926 05:42:29.230758 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.230761 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.230986 (+ 225us) server_negotiation.cc:407] Connection header received
0926 05:42:29.231450 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.231451 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.231465 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.231518 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:29.232418 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.232920 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.233869 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.234072 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.234128 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.236215 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:29.236216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:29.236216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:29.236222 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:29.237394 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.237874 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.237875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.237876 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.237926 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.238155 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.238155 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.238156 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.238253 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:29.238301 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.238575 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.238661 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:42:29.238821 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:42:29.238842 2370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.230697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35885 (local address 127.23.213.193:34303)
0926 05:42:29.230860 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.230897 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.231369 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.231571 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.231575 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.232137 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:29.232332 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.232336 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.232999 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.233000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.233803 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.233805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.234146 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.234146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.234246 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.234791 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:29.234800 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:29.236129 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:29.237478 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:29.237485 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:29.237486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:29.237700 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:29.238005 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:29.238006 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:29.238006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:29.238105 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:29.238304 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:29.238308 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:29.238417 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.238528 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.238644 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
I20260926 05:42:29.239020 24407 external_mini_cluster.cc:1428] Setting key e965ce48b3e3aea39dca9c4996044aea
I20260926 05:42:29.239509 2368 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35885
I20260926 05:42:29.239602 2368 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:29.239804 2368 heartbeater.cc:511] Master 127.23.213.254:35885 requested a full tablet report, sending...
I20260926 05:42:29.240360 2176 ts_manager.cc:194] Registered new tserver with Master: a9356417d2b34dd0923e23759a7982c3 (127.23.213.193:42689)
I20260926 05:42:29.241134 2176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:34303
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:42:29.264889 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35885
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:29.346628 2373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:29.346853 2373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:29.346877 2373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:29.348047 2373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:29.348093 2373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:29.348120 2373 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:42:29.349691 2373 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:29.349960 2373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:29.350214 2373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:29.355374 2381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:29.355393 2379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:29.355443 2378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:29.355641 2373 server_base.cc:1034] running on GCE node
I20260926 05:42:29.355984 2373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:29.356196 2373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:29.357360 2373 hybrid_clock.cc:648] HybridClock initialized: now 1790401349357335 us; error 40 us; skew 500 ppm
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401349, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:29.359920 2373 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:42:29.360407 2373 webserver.cc:460] Webserver started at http://127.23.213.194:46691/ using document root <none> and password file <none>
I20260926 05:42:29.360653 2373 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:29.360723 2373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:29.360877 2373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:29.362042 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "6d83768832064a7d81aecb81ecb48251"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "6ee72f984b57cef2e9228c4b96ec4f9a"
server_key_iv: "31f4125004da7ceb3e9d36345eb203a8"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.362396 2373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "6d83768832064a7d81aecb81ecb48251"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "6ee72f984b57cef2e9228c4b96ec4f9a"
server_key_iv: "31f4125004da7ceb3e9d36345eb203a8"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.363818 2373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:42:29.364537 2388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.364743 2373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:29.364877 2373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "6d83768832064a7d81aecb81ecb48251"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "6ee72f984b57cef2e9228c4b96ec4f9a"
server_key_iv: "31f4125004da7ceb3e9d36345eb203a8"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.364961 2373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:29.387089 2373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:29.389222 2373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:29.389422 2373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:29.389712 2373 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:29.390200 2373 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:29.390249 2373 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.390278 2373 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:29.390296 2373 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.397280 2373 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:41947
I20260926 05:42:29.397320 2501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:41947 every 8 connection(s)
I20260926 05:42:29.397727 2373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:42:29.400134 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 2373
I20260926 05:42:29.400223 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:42:29.400449 24407 external_mini_cluster.cc:1428] Setting key 44cd05b2617de4d8c308a661bcc665b0
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401349, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:29.408361 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.398576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:47723 (local address 127.23.213.254:35885)
0926 05:42:29.398650 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.398652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.398860 (+ 208us) server_negotiation.cc:407] Connection header received
0926 05:42:29.399335 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.399336 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.399348 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.399421 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:29.400450 (+ 1029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.401251 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.402326 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.402643 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.402716 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.405411 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:29.405412 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:29.405413 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:29.405421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:29.406917 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.407347 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.407349 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.407349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.407410 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.407689 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.407690 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.407690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.407810 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:29.407885 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.408179 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.408265 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:42:29.408576 2504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.398544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35885 (local address 127.23.213.194:47723)
0926 05:42:29.398722 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.398765 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.399242 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.399496 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.399499 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.400088 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:29.400350 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.400355 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.401328 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.401329 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.402234 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.402237 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.402803 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.402805 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.402938 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.403587 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:29.403596 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:29.405310 (+ 1714us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:29.407028 (+ 1718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:29.407036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:29.407039 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:29.407268 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:29.407495 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:29.407496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:29.407497 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:29.407613 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:29.407876 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:29.407882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:29.408049 (+ 167us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.408160 (+ 111us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.408326 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":109,"threads_started":1}
I20260926 05:42:29.409054 2502 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35885
I20260926 05:42:29.409191 2502 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:29.409411 2502 heartbeater.cc:511] Master 127.23.213.254:35885 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
I20260926 05:42:29.410396 2176 ts_manager.cc:194] Registered new tserver with Master: 6d83768832064a7d81aecb81ecb48251 (127.23.213.194:41947)
I20260926 05:42:29.410830 2176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:47723
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:42:29.429874 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35885
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:36359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:42:29.513142 2507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:42:29.513417 2507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:42:29.513459 2507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:42:29.514643 2507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:42:29.514722 2507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:42:29.514768 2507 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:42:29.516250 2507 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:36359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:35885
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:42:29.516537 2507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:42:29.516785 2507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:42:29.521906 2515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:29.521931 2512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:42:29.521906 2513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:42:29.521932 2507 server_base.cc:1034] running on GCE node
I20260926 05:42:29.522360 2507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:42:29.522604 2507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:42:29.523757 2507 hybrid_clock.cc:648] HybridClock initialized: now 1790401349523740 us; error 40 us; skew 500 ppm
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401349, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:42:29.526404 2507 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:42:29.526860 2507 webserver.cc:460] Webserver started at http://127.23.213.195:43045/ using document root <none> and password file <none>
I20260926 05:42:29.527045 2507 fs_manager.cc:359] Metadata directory not provided
I20260926 05:42:29.527086 2507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:42:29.527174 2507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:42:29.528177 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "9ec6d264d5814aff912282904118b5d0"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "baafd9c935ac623c2306065b6b943da4"
server_key_iv: "74e4980870eac51c29c134a686f8def3"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.528484 2507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "9ec6d264d5814aff912282904118b5d0"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "baafd9c935ac623c2306065b6b943da4"
server_key_iv: "74e4980870eac51c29c134a686f8def3"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.529739 2507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:42:29.530493 2522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.530660 2507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:42:29.530766 2507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "9ec6d264d5814aff912282904118b5d0"
format_stamp: "Formatted at 2026-09-26 05:42:29 on dist-test-slave-1fdt"
server_key: "baafd9c935ac623c2306065b6b943da4"
server_key_iv: "74e4980870eac51c29c134a686f8def3"
server_key_version: "encryptionkey@0"
I20260926 05:42:29.530845 2507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:42:29.546734 2507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:42:29.549263 2507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:42:29.549443 2507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:42:29.549705 2507 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:42:29.550076 2507 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:42:29.550132 2507 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.550166 2507 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:42:29.550196 2507 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:42:29.556233 2507 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:42919
I20260926 05:42:29.556277 2635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:42919 every 8 connection(s)
I20260926 05:42:29.556614 2507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401349, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:42:29.565150 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 2507
I20260926 05:42:29.565275 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:42:29.565495 24407 external_mini_cluster.cc:1428] Setting key 9085f3e31f864816092c2c7141be178e
I20260926 05:42:29.566488 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.557374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:39591 (local address 127.23.213.254:35885)
0926 05:42:29.557438 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.557440 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.557634 (+ 194us) server_negotiation.cc:407] Connection header received
0926 05:42:29.558279 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.558280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.558290 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.558343 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:29.559276 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.559763 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.560808 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.561019 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.561093 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.563167 (+ 2074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:29.563168 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:29.563168 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:29.563195 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:29.564707 (+ 1512us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.565235 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.565236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.565237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.565300 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.565591 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.565591 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.565592 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.565775 (+ 183us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:29.565901 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.566285 (+ 384us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.566360 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:42:29.566663 2638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.557332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:35885 (local address 127.23.213.195:39591)
0926 05:42:29.557495 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.557539 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.558186 (+ 647us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.558413 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.558417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.558999 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:42:29.559196 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.559201 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.559870 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.559871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.560710 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.560712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.561102 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.561104 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.561234 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.561708 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:42:29.561716 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:42:29.563073 (+ 1357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:42:29.564775 (+ 1702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:29.564780 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:29.564782 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:29.565077 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:29.565373 (+ 296us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:42:29.565374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:42:29.565374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:42:29.565518 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:42:29.565931 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:42:29.565944 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:42:29.566138 (+ 194us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.566253 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.566427 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 05:42:29.567101 2636 heartbeater.cc:347] Connected to a master server at 127.23.213.254:35885
I20260926 05:42:29.567210 2636 heartbeater.cc:464] Registering TS with master...
I20260926 05:42:29.567391 2636 heartbeater.cc:511] Master 127.23.213.254:35885 requested a full tablet report, sending...
I20260926 05:42:29.567806 2176 ts_manager.cc:194] Registered new tserver with Master: 9ec6d264d5814aff912282904118b5d0 (127.23.213.195:42919)
I20260926 05:42:29.568138 2176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:39591
I20260926 05:42:29.577627 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:42:29.585054 2234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.579008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53448 (local address 127.23.213.254:35885)
0926 05:42:29.579087 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.579089 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.579176 (+ 87us) server_negotiation.cc:407] Connection header received
0926 05:42:29.579217 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.579218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.579226 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.579278 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:42:29.579843 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.580320 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.581014 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.581200 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.581256 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.582485 (+ 1229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:42:29.582486 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:42:29.582486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:42:29.582493 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:42:29.584088 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.584393 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.584394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.584394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.584466 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:42:29.584628 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:42:29.584629 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:42:29.584629 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:42:29.584768 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:42:29.584872 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.584927 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.584986 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:42:29.586886 2176 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53448:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:42:29.588495 2176 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:42:29.594933 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.590841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:42919 (local address 127.0.0.1:37388)
0926 05:42:29.590986 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.590997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.591051 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.591523 (+ 472us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.591524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.591526 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:29.591580 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.591582 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.592583 (+ 1001us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.592584 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.593645 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.593647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.594610 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.594611 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.594689 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.594704 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.594797 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.594854 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":89,"threads_started":1}
I20260926 05:42:29.594946 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.591163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.23.213.195:42919)
0926 05:42:29.591353 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.591358 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.591367 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:29.591404 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.591406 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.591421 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.591675 (+ 254us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:29.591740 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.592478 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.593762 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.594525 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.594615 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.594675 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.594816 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.594857 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":88,"threads_started":1}
I20260926 05:42:29.595019 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.590997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40794 (local address 127.23.213.193:42689)
0926 05:42:29.591197 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.591201 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.591209 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:29.591241 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.591243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.591276 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.591360 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:29.591575 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.592725 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.593941 (+ 1216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.594547 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.594626 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.594679 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.594890 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.594934 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260926 05:42:29.595219 2647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.590803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:42689 (local address 127.0.0.1:40794)
0926 05:42:29.590930 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.590975 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.591037 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.591387 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.591390 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.591393 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:29.591469 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.591472 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.592841 (+ 1369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.592842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.593823 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.593826 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.594712 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.594713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.594785 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.594789 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.595097 (+ 308us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.595152 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260926 05:42:29.595503 2650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.591028 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:41947 (local address 127.0.0.1:52832)
0926 05:42:29.591493 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.591503 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.591543 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.591930 (+ 387us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.591931 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.591933 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:29.591974 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.591976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.593212 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.593213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.594298 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.594301 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.595257 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.595258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.595342 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.595345 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.595398 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.595441 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":68,"threads_started":1}
I20260926 05:42:29.595793 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.591431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52832 (local address 127.23.213.194:41947)
0926 05:42:29.591702 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.591706 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.591715 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:42:29.591752 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.591755 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.591769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.591859 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:29.592046 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.593104 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.594399 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.595111 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.595176 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.595222 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.595676 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.595715 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":238,"thread_start_us":81,"threads_started":1}
I20260926 05:42:29.595871 2570 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfd50e5f9c041f19351c341d8229682 (DEFAULT_TABLE table=test-table [id=f1903b8dad0048bca54773196ec2506e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:29.595871 2302 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfd50e5f9c041f19351c341d8229682 (DEFAULT_TABLE table=test-table [id=f1903b8dad0048bca54773196ec2506e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:29.596230 2302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfd50e5f9c041f19351c341d8229682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:29.596231 2570 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfd50e5f9c041f19351c341d8229682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:29.596449 2436 tablet_service.cc:1467] Processing CreateTablet for tablet 9dfd50e5f9c041f19351c341d8229682 (DEFAULT_TABLE table=test-table [id=f1903b8dad0048bca54773196ec2506e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:42:29.596704 2436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9dfd50e5f9c041f19351c341d8229682. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:42:29.599326 2653 tablet_bootstrap.cc:492] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Bootstrap starting.
I20260926 05:42:29.599825 2654 tablet_bootstrap.cc:492] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Bootstrap starting.
I20260926 05:42:29.599874 2655 tablet_bootstrap.cc:492] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3: Bootstrap starting.
I20260926 05:42:29.600049 2653 tablet_bootstrap.cc:654] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:29.600482 2654 tablet_bootstrap.cc:654] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:29.600490 2655 tablet_bootstrap.cc:654] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3: Neither blocks nor log segments found. Creating new log.
I20260926 05:42:29.600634 2653 log.cc:824] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:29.601083 2655 log.cc:824] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:29.601083 2654 log.cc:824] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Log is configured to *not* fsync() on all Append() calls
I20260926 05:42:29.601454 2653 tablet_bootstrap.cc:492] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: No bootstrap required, opened a new log
I20260926 05:42:29.601538 2653 ts_tablet_manager.cc:1397] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:42:29.601907 2654 tablet_bootstrap.cc:492] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: No bootstrap required, opened a new log
I20260926 05:42:29.601979 2654 ts_tablet_manager.cc:1397] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:42:29.602083 2655 tablet_bootstrap.cc:492] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3: No bootstrap required, opened a new log
I20260926 05:42:29.602139 2655 ts_tablet_manager.cc:1397] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:42:29.603219 2653 raft_consensus.cc:348] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.603382 2653 raft_consensus.cc:374] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:29.603425 2653 raft_consensus.cc:729] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d83768832064a7d81aecb81ecb48251, State: Initialized, Role: FOLLOWER
I20260926 05:42:29.603569 2653 consensus_queue.cc:260] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.603853 2653 ts_tablet_manager.cc:1428] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:42:29.604017 2502 heartbeater.cc:503] Master 127.23.213.254:35885 was elected leader, sending a full tablet report...
I20260926 05:42:29.604064 2654 raft_consensus.cc:348] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.604064 2655 raft_consensus.cc:348] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.604224 2655 raft_consensus.cc:374] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:29.604224 2654 raft_consensus.cc:374] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:42:29.604259 2655 raft_consensus.cc:729] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9356417d2b34dd0923e23759a7982c3, State: Initialized, Role: FOLLOWER
I20260926 05:42:29.604260 2654 raft_consensus.cc:729] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec6d264d5814aff912282904118b5d0, State: Initialized, Role: FOLLOWER
I20260926 05:42:29.604391 2655 consensus_queue.cc:260] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.604391 2654 consensus_queue.cc:260] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.604666 2655 ts_tablet_manager.cc:1428] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:42:29.604666 2654 ts_tablet_manager.cc:1428] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:42:29.604820 2368 heartbeater.cc:503] Master 127.23.213.254:35885 was elected leader, sending a full tablet report...
I20260926 05:42:29.605134 2636 heartbeater.cc:503] Master 127.23.213.254:35885 was elected leader, sending a full tablet report...
W20260926 05:42:29.648372 2503 tablet.cc:2369] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:42:29.730705 2369 tablet.cc:2369] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:42:29.795642 2661 raft_consensus.cc:482] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:42:29.795789 2661 raft_consensus.cc:504] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.796212 2661 leader_election.cc:288] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9ec6d264d5814aff912282904118b5d0 (127.23.213.195:42919), 6d83768832064a7d81aecb81ecb48251 (127.23.213.194:41947)
I20260926 05:42:29.799446 2652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.796728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:50803 (local address 127.23.213.194:41947)
0926 05:42:29.796783 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.796785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.796855 (+ 70us) server_negotiation.cc:407] Connection header received
0926 05:42:29.796924 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.796925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.796941 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.796985 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:29.797180 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.797682 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.798682 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.799090 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.799172 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.799182 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.799332 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.799375 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:42:29.799451 2663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.796658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:41947 (local address 127.23.213.193:50803)
0926 05:42:29.796776 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.796786 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.796846 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.797017 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.797018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.797020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:29.797098 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.797100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.797770 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.797771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.798584 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.798586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.799173 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.799174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.799244 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.799248 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.799301 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.799353 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":49,"threads_started":1}
I20260926 05:42:29.799486 2651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.796531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:32807 (local address 127.23.213.195:42919)
0926 05:42:29.796617 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.796620 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.796713 (+ 93us) server_negotiation.cc:407] Connection header received
0926 05:42:29.796920 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.796921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.796933 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.797021 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:42:29.797237 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.797681 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.798690 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.799094 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.799182 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.799190 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.799378 (+ 188us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.799420 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 05:42:29.799468 2662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.796415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:42919 (local address 127.23.213.193:32807)
0926 05:42:29.796617 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.796629 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:42:29.796731 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:42:29.797050 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:42:29.797051 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:42:29.797053 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:42:29.797155 (+ 102us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.797158 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.797813 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.797814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.798581 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:42:29.798583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.799200 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.799201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:42:29.799269 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.799272 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:42:29.799353 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:42:29.799389 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":107,"threads_started":1}
I20260926 05:42:29.799836 2456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd50e5f9c041f19351c341d8229682" candidate_uuid: "a9356417d2b34dd0923e23759a7982c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d83768832064a7d81aecb81ecb48251" is_pre_election: true
I20260926 05:42:29.800001 2456 raft_consensus.cc:2454] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9356417d2b34dd0923e23759a7982c3 in term 0.
I20260926 05:42:29.800067 2590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd50e5f9c041f19351c341d8229682" candidate_uuid: "a9356417d2b34dd0923e23759a7982c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec6d264d5814aff912282904118b5d0" is_pre_election: true
I20260926 05:42:29.800256 2590 raft_consensus.cc:2454] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a9356417d2b34dd0923e23759a7982c3 in term 0.
I20260926 05:42:29.800257 2258 leader_election.cc:302] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6d83768832064a7d81aecb81ecb48251, a9356417d2b34dd0923e23759a7982c3; no voters:
I20260926 05:42:29.800386 2661 raft_consensus.cc:2781] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:42:29.800468 2661 raft_consensus.cc:482] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:42:29.800509 2661 raft_consensus.cc:3037] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:29.801231 2661 raft_consensus.cc:504] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.801378 2661 leader_election.cc:288] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [CANDIDATE]: Term 1 election: Requested vote from peers 9ec6d264d5814aff912282904118b5d0 (127.23.213.195:42919), 6d83768832064a7d81aecb81ecb48251 (127.23.213.194:41947)
I20260926 05:42:29.801554 2590 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd50e5f9c041f19351c341d8229682" candidate_uuid: "a9356417d2b34dd0923e23759a7982c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9ec6d264d5814aff912282904118b5d0"
I20260926 05:42:29.801581 2456 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9dfd50e5f9c041f19351c341d8229682" candidate_uuid: "a9356417d2b34dd0923e23759a7982c3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d83768832064a7d81aecb81ecb48251"
I20260926 05:42:29.801646 2590 raft_consensus.cc:3037] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:29.801657 2456 raft_consensus.cc:3037] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:42:29.802390 2590 raft_consensus.cc:2454] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9356417d2b34dd0923e23759a7982c3 in term 1.
I20260926 05:42:29.802531 2456 raft_consensus.cc:2454] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a9356417d2b34dd0923e23759a7982c3 in term 1.
I20260926 05:42:29.802551 2258 leader_election.cc:302] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ec6d264d5814aff912282904118b5d0, a9356417d2b34dd0923e23759a7982c3; no voters:
I20260926 05:42:29.802647 2661 raft_consensus.cc:2781] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:42:29.802814 2661 raft_consensus.cc:686] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [term 1 LEADER]: Becoming Leader. State: Replica: a9356417d2b34dd0923e23759a7982c3, State: Running, Role: LEADER
I20260926 05:42:29.802953 2661 consensus_queue.cc:237] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } }
I20260926 05:42:29.803745 2177 catalog_manager.cc:5495] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a9356417d2b34dd0923e23759a7982c3 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "a9356417d2b34dd0923e23759a7982c3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a9356417d2b34dd0923e23759a7982c3" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42689 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 } health_report { overall_health: UNKNOWN } } }
W20260926 05:42:29.807272 2637 tablet.cc:2369] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:42:29.820704 2649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.818555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.23.213.193:42689)
0926 05:42:29.818599 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:42:29.818601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:42:29.818609 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:42:29.818652 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:42:29.818653 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:42:29.818672 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:42:29.818735 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:42:29.818849 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.819325 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.820040 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:42:29.820225 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:42:29.820305 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:42:29.820408 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:42:29.820536 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:42:29.820594 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:42:29.820626 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 05:42:29.820643 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:42:29.822679 2456 raft_consensus.cc:1261] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 1 FOLLOWER]: Refusing update from remote peer a9356417d2b34dd0923e23759a7982c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:29.822690 2590 raft_consensus.cc:1261] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 1 FOLLOWER]: Refusing update from remote peer a9356417d2b34dd0923e23759a7982c3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:42:29.823077 2664 consensus_queue.cc:1035] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d83768832064a7d81aecb81ecb48251" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41947 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:29.823220 2661 consensus_queue.cc:1035] T 9dfd50e5f9c041f19351c341d8229682 P a9356417d2b34dd0923e23759a7982c3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ec6d264d5814aff912282904118b5d0" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:42:29.825630 2667 mvcc.cc:205] Tried to move back new op lower bound from 7333483928870936576 to 7333483928793448448. Current Snapshot: MvccSnapshot[applied={T|T < 7333483928870936576}]
I20260926 05:42:29.827841 2669 mvcc.cc:205] Tried to move back new op lower bound from 7333483928870936576 to 7333483928793448448. Current Snapshot: MvccSnapshot[applied={T|T < 7333483928870936576 or (T in {7333483928870936576})}]
I20260926 05:42:29.827955 2177 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:29.828049 2177 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53448:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:42:29.828814 2177 catalog_manager.cc:5782] T 00000000000000000000000000000000 P abaf72dd3dc644e3a72b35c72bdbb800: Sending DeleteTablet for 3 replicas of tablet 9dfd50e5f9c041f19351c341d8229682
I20260926 05:42:29.829147 2302 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfd50e5f9c041f19351c341d8229682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:29 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:40794
I20260926 05:42:29.829181 2436 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfd50e5f9c041f19351c341d8229682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:29 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:52832
I20260926 05:42:29.829180 2570 tablet_service.cc:1514] Processing DeleteTablet for tablet 9dfd50e5f9c041f19351c341d8229682 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:42:29 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37388
I20260926 05:42:29.829541 2678 tablet_replica.cc:330] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: stopping tablet replica
I20260926 05:42:29.829659 2678 raft_consensus.cc:2229] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:29.829777 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 2239
I20260926 05:42:29.829815 2678 raft_consensus.cc:2258] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:29.830511 2677 tablet_replica.cc:330] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: stopping tablet replica
I20260926 05:42:29.830598 2677 raft_consensus.cc:2229] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:42:29.830669 2677 raft_consensus.cc:2258] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:42:29.831050 2677 ts_tablet_manager.cc:1905] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:29.831511 2678 ts_tablet_manager.cc:1905] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:42:29.832768 2677 ts_tablet_manager.cc:1918] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:29.832911 2677 log.cc:1196] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/9dfd50e5f9c041f19351c341d8229682
I20260926 05:42:29.833180 2678 ts_tablet_manager.cc:1918] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:42:29.833264 2678 log.cc:1196] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/9dfd50e5f9c041f19351c341d8229682
I20260926 05:42:29.833263 2677 ts_tablet_manager.cc:1939] T 9dfd50e5f9c041f19351c341d8229682 P 9ec6d264d5814aff912282904118b5d0: Deleting consensus metadata
I20260926 05:42:29.833552 2678 ts_tablet_manager.cc:1939] T 9dfd50e5f9c041f19351c341d8229682 P 6d83768832064a7d81aecb81ecb48251: Deleting consensus metadata
I20260926 05:42:29.833693 2164 catalog_manager.cc:4841] TS 9ec6d264d5814aff912282904118b5d0 (127.23.213.195:42919): tablet 9dfd50e5f9c041f19351c341d8229682 (table test-table [id=f1903b8dad0048bca54773196ec2506e]) successfully deleted
I20260926 05:42:29.834053 2164 catalog_manager.cc:4841] TS 6d83768832064a7d81aecb81ecb48251 (127.23.213.194:41947): tablet 9dfd50e5f9c041f19351c341d8229682 (table test-table [id=f1903b8dad0048bca54773196ec2506e]) successfully deleted
W20260926 05:42:29.835891 2163 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:42689 (error 108)
I20260926 05:42:29.836436 2648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:42:29.836298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:42689 (local address 127.0.0.1:40806)
0926 05:42:29.836354 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:42:29.836396 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:42689: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:42:29.836517 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 2373
W20260926 05:42:29.836587 2163 catalog_manager.cc:4584] TS a9356417d2b34dd0923e23759a7982c3 (127.23.213.193:42689): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9dfd50e5f9c041f19351c341d8229682: Network error: Client connection negotiation failed: client connection to 127.23.213.193:42689: connect: Connection refused (error 111)
I20260926 05:42:29.841346 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 2507
I20260926 05:42:29.849174 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 2145
2026-09-26T05:42:29Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2087 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2682](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2682](info): set up 2 sockets
Sep 26 05:42:29 dist-test-slave-1fdt krb5kdc[2682](info): commencing operation
krb5kdc: starting...
W20260926 05:42:30.894799 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:42:30 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401350, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:42:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:42:30Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/rangeradmin_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/rangeradmin_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/rangerlookup_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/rangerlookup_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
I20260926 05:42:31.064493 24407 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPYhTSt/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/postgres -l logfile start
2026-09-26 05:42:33.640 UTC [2712] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 05:42:33.640 UTC [2712] LOG: listening on IPv4 address "127.23.213.212", port 50337
2026-09-26 05:42:33.644 UTC [2712] LOG: listening on Unix socket "/tmp/.s.PGSQL.50337"
2026-09-26 05:42:33.666 UTC [2715] LOG: database system was shut down at 2026-09-26 05:42:32 UTC
2026-09-26 05:42:33.670 UTC [2712] LOG: database system is ready to accept connections
I20260926 05:42:34.618688 24407 mini_postgres.cc:96] Postgres bound to 50337
2026-09-26 05:42:34.624 UTC [2723] FATAL: database "slave" does not exist
127.23.213.212:50337 - accepting connections
I20260926 05:42:34.625272 24407 mini_ranger.cc:161] Starting Ranger...
I20260926 05:42:34.640055 24407 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 05:42:35.153549 24407 mini_ranger.cc:87] Created ranger Postgres database
I20260926 05:42:35.153636 24407 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-admin
2026-09-26 05:42:35,443 [I] DB FLAVOR :POSTGRES
2026-09-26 05:42:35,443 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:42:35,443 [I] Checking connection..
2026-09-26 05:42:35,443 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 05:42:35,709 [I] Checking connection passed.
2026-09-26 05:42:35,710 [I] --------- Verifying version history table ---------
2026-09-26 05:42:35,710 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:42:35,960 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:42:35,960 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:42:36,218 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:42:36,218 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 05:42:36,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 05:42:36.447 UTC [2787] WARNING: there is no transaction in progress
2026-09-26 05:42:36,454 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 05:42:36,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:42:36,707 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 05:42:36,708 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:42:36,708 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 05:42:36,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 05:42:37,179 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1fdt.c.gcp-upstream.internal','N') ;"
2026-09-26 05:42:37,402 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:42:37,645 [I] Table x_portal_user does not exist in database ranger
2026-09-26 05:42:37,646 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:42:37,897 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 05:42:37,898 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:42:38,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:42:38,431 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 05:42:38,432 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 05:42:39.471 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.488 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.506 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.522 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.538 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.556 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.612 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.619 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.634 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.644 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.655 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.669 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.679 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.685 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.692 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:39.700 UTC [2895] WARNING: there is no transaction in progress
2026-09-26 05:42:40,377 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 05:42:40,378 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 05:42:40,625 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 05:42:40,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:42:40,875 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 05:42:40,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:42:41,114 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 05:42:41,115 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:42:41,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:42:41,596 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1fdt.c.gcp-upstream.internal';"
2026-09-26 05:42:41,826 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 05:42:41,827 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 05:42:42,068 [I] DB_PATCHES have already been applied
I20260926 05:42:42.074422 24407 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPYhTSt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 05:42:42.074508 24407 mini_ranger.cc:193] Using host: 127.23.213.212
I20260926 05:42:42.077940 24407 mini_ranger.cc:227] Ranger admin URL: http://127.23.213.212:32773
Sep 26, 2026 5:42:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:42:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:42:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 5:42:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 5:42:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 5:42:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 5:42:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 05:42:43.186323 24407 mini_ranger.cc:160] Time spent starting Ranger: real 8.561s user 0.003s sys 0.006s
Sep 26, 2026 5:43:03 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 5:43:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.MetricsREST
Sep 26, 2026 5:43:03 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 5:43:03 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 5:43:04 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 05:43:05.348791 24407 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/rangerkms_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/rangerkms_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.213.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.213.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 05:43:05 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401385, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 05:43:05.434638 24407 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 05:43:05.443977 24407 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 05:43:06.335433 24407 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 05:43:06.335497 24407 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms
I20260926 05:43:06.335510 24407 mini_ranger_kms.cc:227] Using postgres at 127.23.213.212:50337
2026-09-26 05:43:06,500 [I] DB FLAVOR :POSTGRES
2026-09-26 05:43:06,500 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:43:06,500 [I] Checking connection
2026-09-26 05:43:06,500 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 05:43:06,730 [I] connection success
2026-09-26 05:43:06,730 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 05:43:06,730 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 05:43:06,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 05:43:06,969 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 05:43:06,969 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:43:06,969 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 05:43:06,969 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:50337/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 05:43:07,218 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 05:43:07.659876 24407 mini_ranger_kms.cc:310] Created kms service
I20260926 05:43:07.861830 24407 mini_ranger_kms.cc:326] Created kudu user
I20260926 05:43:07.901786 24407 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 05:43:07 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401385, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:08.314834 24407 mini_ranger_kms.cc:384] Added ranger policy
I20260926 05:43:08.314946 24407 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 05:43:08.314970 24407 mini_ranger_kms.cc:240] Using host: 127.23.213.212
I20260926 05:43:08.318765 24407 mini_ranger_kms.cc:276] Ranger KMS PID: 3075
I20260926 05:43:08.318883 24407 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.213.212:36133
Sep 26, 2026 5:43:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:43:09 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:43:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 5:43:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 5:43:09 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 5:43:09 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 05:43:10.570426 24407 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.136s user 0.008s sys 0.139s
I20260926 05:43:10.570538 24407 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 05:43:10.570623 24407 mini_ranger_kms.cc:401] 127.23.213.212:36133/kms/v1/keys
Sep 26 05:43:10 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401390, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.213.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 05:43:11 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401390, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.213.212@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
Sep 26 05:43:11 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401390, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.213.212@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
Sep 26 05:43:13 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:43:13.612651 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41995
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:41995
--ranger_config_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:36133/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:43:13.698019 3126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:43:13.698240 3126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:43:13.698261 3126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:43:13.699458 3126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:43:13.699503 3126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:43:13.699517 3126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:43:13.699530 3126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:43:13.699542 3126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:43:13.701232 3126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.213.212:36133/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:41995
--ranger_config_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:43:13.701493 3126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:43:13.701763 3126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:43:13.707170 3131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:13.707283 3126 server_base.cc:1034] running on GCE node
W20260926 05:43:13.707252 3132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:43:13.707155 3134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:13.707742 3126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:43:13.708063 3126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:43:13.709235 3126 hybrid_clock.cc:648] HybridClock initialized: now 1790401393709221 us; error 34 us; skew 500 ppm
Sep 26 05:43:13 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401393, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:43:13.715451 3126 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:43:13.715967 3126 webserver.cc:460] Webserver started at http://127.23.213.254:35275/ using document root <none> and password file <none>
I20260926 05:43:13.716212 3126 fs_manager.cc:359] Metadata directory not provided
I20260926 05:43:13.716276 3126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:43:13.716444 3126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:43:13 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401393, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:13.777954 3126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "4648d28ba55b49d5baac032b2f7362a6"
format_stamp: "Formatted at 2026-09-26 05:43:13 on dist-test-slave-1fdt"
server_key: "d517857acf4ad891d5ed655ddce7c4b4"
server_key_iv: "7d4611e6ec3fb6afe35bd28ef34681bf"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:13.778374 3126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "4648d28ba55b49d5baac032b2f7362a6"
format_stamp: "Formatted at 2026-09-26 05:43:13 on dist-test-slave-1fdt"
server_key: "d517857acf4ad891d5ed655ddce7c4b4"
server_key_iv: "7d4611e6ec3fb6afe35bd28ef34681bf"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:13.779879 3126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:43:13.794631 3141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:13.794811 3126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:43:13.794896 3126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "4648d28ba55b49d5baac032b2f7362a6"
format_stamp: "Formatted at 2026-09-26 05:43:13 on dist-test-slave-1fdt"
server_key: "d517857acf4ad891d5ed655ddce7c4b4"
server_key_iv: "7d4611e6ec3fb6afe35bd28ef34681bf"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:13.795070 3126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:43:13.819682 3126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:43:13.822154 3126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:43:13.822343 3126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:43:13.827522 3126 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:41995
I20260926 05:43:13.827585 3193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:41995 every 8 connection(s)
I20260926 05:43:13.827980 3126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:43:13.828750 3194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:43:13.829265 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 3126
I20260926 05:43:13.829370 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:43:13.829551 24407 external_mini_cluster.cc:1428] Setting key ff3daf50e560f2bbffc74f77f6cdee9e
I20260926 05:43:13.831526 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6: Bootstrap starting.
I20260926 05:43:13.832266 3194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6: Neither blocks nor log segments found. Creating new log.
I20260926 05:43:13.832587 3194 log.cc:824] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:43:13.833343 3194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6: No bootstrap required, opened a new log
I20260926 05:43:13.835677 3194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648d28ba55b49d5baac032b2f7362a6" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41995 } }
I20260926 05:43:13.835839 3194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:43:13.835901 3194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4648d28ba55b49d5baac032b2f7362a6, State: Initialized, Role: FOLLOWER
I20260926 05:43:13.836071 3194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648d28ba55b49d5baac032b2f7362a6" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41995 } }
I20260926 05:43:13.836160 3194 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:43:13.836194 3194 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:43:13.836261 3194 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:43:13.836988 3194 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648d28ba55b49d5baac032b2f7362a6" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41995 } }
I20260926 05:43:13.837107 3194 leader_election.cc:302] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4648d28ba55b49d5baac032b2f7362a6; no voters:
I20260926 05:43:13.837308 3194 leader_election.cc:288] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:43:13.837337 3199 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:43:13.839031 3194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:43:13.839238 3199 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [term 1 LEADER]: Becoming Leader. State: Replica: 4648d28ba55b49d5baac032b2f7362a6, State: Running, Role: LEADER
I20260926 05:43:13.839442 3199 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648d28ba55b49d5baac032b2f7362a6" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41995 } }
Sep 26 05:43:13 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401393, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:43:13.840643 3200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4648d28ba55b49d5baac032b2f7362a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648d28ba55b49d5baac032b2f7362a6" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41995 } } }
I20260926 05:43:13.840802 3200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [sys.catalog]: This master's current role is: LEADER
I20260926 05:43:13.840874 3194 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 05:43:13.840982 3200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4648d28ba55b49d5baac032b2f7362a6. Latest consensus state: current_term: 1 leader_uuid: "4648d28ba55b49d5baac032b2f7362a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4648d28ba55b49d5baac032b2f7362a6" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41995 } } }
I20260926 05:43:13.841046 3200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6 [sys.catalog]: This master's current role is: LEADER
I20260926 05:43:13.841490 3213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:43:13.842088 3213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:43:13.846995 3197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:13.830136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47440 (local address 127.23.213.254:41995)
0926 05:43:13.831724 (+ 1588us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:13.831734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:13.831770 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:43:13.832387 (+ 617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:13.832399 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:13.832473 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:13.832652 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:43:13.833488 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:13.834995 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:13.836088 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:13.836719 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:13.836846 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:13.840224 (+ 3378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:43:13.840229 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:43:13.840229 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:43:13.840243 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:43:13.842374 (+ 2131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:13.842793 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:13.842796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:13.842796 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:13.842901 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:13.844153 (+ 1252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:13.844186 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:13.844187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:13.845540 (+ 1353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:43:13.845899 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:13.846024 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 05:43:13.846555 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1512,"thread_start_us":77,"threads_started":1}
I20260926 05:43:13.850260 3213 catalog_manager.cc:1347] Generated new cluster ID: 34be65dc5ca248c1a2e9d7e85d0c32b5
I20260926 05:43:13.850313 3213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:43:13.858304 3213 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:43:13.859237 3213 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:43:13.871915 3213 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6: Generated new TSK 0
I20260926 05:43:13.872263 3213 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:43:14.649134 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:36133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41995
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:43:14.770143 3239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:43:14.770435 3239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:43:14.770470 3239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:43:14.772158 3239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:43:14.772222 3239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:43:14.772293 3239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:43:14.774536 3239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.213.212:36133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3239
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:43:14.774802 3239 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:43:14.775068 3239 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:43:14.780539 3245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:43:14.780546 3247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:43:14.780678 3244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:14.780845 3239 server_base.cc:1034] running on GCE node
I20260926 05:43:14.781900 3239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:43:14.782205 3239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:43:14.785518 3239 hybrid_clock.cc:648] HybridClock initialized: now 1790401394785493 us; error 64 us; skew 500 ppm
Sep 26 05:43:14 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:43:14.788180 3239 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:43:14.788682 3239 webserver.cc:460] Webserver started at http://127.23.213.193:37189/ using document root <none> and password file <none>
I20260926 05:43:14.788932 3239 fs_manager.cc:359] Metadata directory not provided
I20260926 05:43:14.789000 3239 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:43:14.789168 3239 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:43:14 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:14.807235 3239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "a474cdb8ab614db1a177e1cf8a3525db"
format_stamp: "Formatted at 2026-09-26 05:43:14 on dist-test-slave-1fdt"
server_key: "d173eb38f408865ef58fa5cf818915cb"
server_key_iv: "db136c147d1af0aee1a93676d715da49"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:14.807677 3239 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "a474cdb8ab614db1a177e1cf8a3525db"
format_stamp: "Formatted at 2026-09-26 05:43:14 on dist-test-slave-1fdt"
server_key: "d173eb38f408865ef58fa5cf818915cb"
server_key_iv: "db136c147d1af0aee1a93676d715da49"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:14.809243 3239 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:43:14.833937 3256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:14.834214 3239 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260926 05:43:14.834318 3239 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "a474cdb8ab614db1a177e1cf8a3525db"
format_stamp: "Formatted at 2026-09-26 05:43:14 on dist-test-slave-1fdt"
server_key: "d173eb38f408865ef58fa5cf818915cb"
server_key_iv: "db136c147d1af0aee1a93676d715da49"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:14.834417 3239 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:43:14.866828 3239 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:43:14.868988 3239 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:43:14.869150 3239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:43:14.869428 3239 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:43:14.869865 3239 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:43:14.869920 3239 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:14.869951 3239 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:43:14.870002 3239 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:14.877291 3239 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:43203
I20260926 05:43:14.877758 3239 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:43:14.878221 3369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:43203 every 8 connection(s)
Sep 26 05:43:14 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:43:14.886054 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 3239
I20260926 05:43:14.886160 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:43:14.886360 24407 external_mini_cluster.cc:1428] Setting key fb59c112de22ac74dfa58fe5aba33fe1
I20260926 05:43:14.892226 3372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:14.878518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41995 (local address 127.23.213.193:51051)
0926 05:43:14.878838 (+ 320us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:14.878884 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:14.879521 (+ 637us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:14.879900 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:14.879905 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:14.880162 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:43:14.880446 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:14.880453 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:14.881509 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:14.881510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:14.882853 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:14.882856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:14.883388 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:14.883389 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:14.883525 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:14.884128 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:43:14.884138 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:43:14.885746 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:43:14.890540 (+ 4794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:43:14.890548 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:43:14.890551 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:43:14.890855 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:43:14.891067 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:43:14.891067 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:43:14.891068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:43:14.891205 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:43:14.891497 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:43:14.891503 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:43:14.891662 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 05:43:14.891798 (+ 136us) client_negotiation.cc:240] Negotiation successful
0926 05:43:14.891960 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":76,"threads_started":1}
I20260926 05:43:14.892974 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:14.878832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:51051 (local address 127.23.213.254:41995)
0926 05:43:14.879695 (+ 863us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:14.879697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:14.879705 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:43:14.879731 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:14.879731 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:14.879743 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:14.880541 (+ 798us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:43:14.880586 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:14.881385 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:14.882955 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:14.883278 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:14.885845 (+ 2567us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:14.888565 (+ 2720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:43:14.888567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:43:14.888567 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:43:14.888574 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:43:14.890436 (+ 1862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:14.890943 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:14.890944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:14.890944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:14.891002 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:14.891283 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:14.891284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:14.891284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:14.891420 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:43:14.892749 (+ 1329us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:14.892792 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 05:43:14.892875 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":60,"threads_started":1}
I20260926 05:43:14.893498 3370 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41995
I20260926 05:43:14.893589 3370 heartbeater.cc:464] Registering TS with master...
I20260926 05:43:14.893864 3370 heartbeater.cc:511] Master 127.23.213.254:41995 requested a full tablet report, sending...
I20260926 05:43:14.894438 3158 ts_manager.cc:194] Registered new tserver with Master: a474cdb8ab614db1a177e1cf8a3525db (127.23.213.193:43203)
I20260926 05:43:14.895306 3158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:51051
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:43:14.932898 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:36133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41995
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:43:14 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 05:43:15.051956 3378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:43:15.052332 3378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:43:15.052412 3378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:43:15.054230 3378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:43:15.054302 3378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:43:15.054344 3378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:43:15.056552 3378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.213.212:36133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:43:15.056828 3378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:43:15.057114 3378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:43:15.063480 3387 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:15.063674 3378 server_base.cc:1034] running on GCE node
W20260926 05:43:15.063476 3385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:43:15.064250 3384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:15.064502 3378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:43:15.064796 3378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:43:15.065973 3378 hybrid_clock.cc:648] HybridClock initialized: now 1790401395065944 us; error 47 us; skew 500 ppm
Sep 26 05:43:15 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:43:15.069324 3378 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:43:15.069931 3378 webserver.cc:460] Webserver started at http://127.23.213.194:39545/ using document root <none> and password file <none>
I20260926 05:43:15.070212 3378 fs_manager.cc:359] Metadata directory not provided
I20260926 05:43:15.070291 3378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:43:15.070525 3378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:43:15 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:15.082473 3378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "d9d8d2cb5b6c45a981545b0bed86919a"
format_stamp: "Formatted at 2026-09-26 05:43:15 on dist-test-slave-1fdt"
server_key: "2b0a520b08a9d981f7d434663341c288"
server_key_iv: "5b3d1b5fbe80a4b82bf84ab43aebd895"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:15.082888 3378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "d9d8d2cb5b6c45a981545b0bed86919a"
format_stamp: "Formatted at 2026-09-26 05:43:15 on dist-test-slave-1fdt"
server_key: "2b0a520b08a9d981f7d434663341c288"
server_key_iv: "5b3d1b5fbe80a4b82bf84ab43aebd895"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:15.086119 3378 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
I20260926 05:43:15.103704 3394 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:15.103968 3378 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260926 05:43:15.104074 3378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "d9d8d2cb5b6c45a981545b0bed86919a"
format_stamp: "Formatted at 2026-09-26 05:43:15 on dist-test-slave-1fdt"
server_key: "2b0a520b08a9d981f7d434663341c288"
server_key_iv: "5b3d1b5fbe80a4b82bf84ab43aebd895"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:15.104174 3378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:43:15.118837 3378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:43:15.121356 3378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:43:15.121567 3378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:43:15.121856 3378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:43:15.122262 3378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:43:15.122356 3378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:15.122426 3378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:43:15.122483 3378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:15.128990 3378 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:36837
I20260926 05:43:15.129505 3378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:43:15.131040 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 3378
I20260926 05:43:15.131162 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:43:15.131373 24407 external_mini_cluster.cc:1428] Setting key 012078212283f3abddfe1e4c196be8a2
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
I20260926 05:43:15.142946 3507 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:36837 every 8 connection(s)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Sep 26 05:43:15 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:43:15.156102 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.142802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:57783 (local address 127.23.213.254:41995)
0926 05:43:15.142850 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.142852 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.143622 (+ 770us) server_negotiation.cc:407] Connection header received
0926 05:43:15.144392 (+ 770us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.144393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.144408 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.144439 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:43:15.147261 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.148078 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.149690 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.150028 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.150073 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.152705 (+ 2632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:43:15.152706 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:43:15.152706 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:43:15.152714 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:43:15.154495 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:15.154988 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:15.154989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:15.154990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:15.155051 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:15.155322 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:15.155323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:15.155323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:15.155459 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:43:15.155500 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.155881 (+ 381us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.155975 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:43:15.156872 3511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.142504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41995 (local address 127.23.213.194:57783)
0926 05:43:15.143488 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.143536 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:15.144259 (+ 723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:15.144636 (+ 377us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:15.144641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:15.146832 (+ 2191us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:43:15.147131 (+ 299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.147139 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.148231 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.148232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.149582 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.149585 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.150170 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.150171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.150315 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.150912 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:43:15.150923 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:43:15.152591 (+ 1668us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:43:15.154589 (+ 1998us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:43:15.154595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:43:15.154598 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:43:15.154897 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:43:15.155119 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:43:15.155120 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:43:15.155121 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:43:15.155249 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:43:15.155544 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:43:15.155549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:43:15.155715 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 05:43:15.156459 (+ 744us) client_negotiation.cc:240] Negotiation successful
0926 05:43:15.156624 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":84,"threads_started":1}
I20260926 05:43:15.157529 3508 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41995
I20260926 05:43:15.157624 3508 heartbeater.cc:464] Registering TS with master...
I20260926 05:43:15.157914 3508 heartbeater.cc:511] Master 127.23.213.254:41995 requested a full tablet report, sending...
I20260926 05:43:15.158275 3158 ts_manager.cc:194] Registered new tserver with Master: d9d8d2cb5b6c45a981545b0bed86919a (127.23.213.194:36837)
I20260926 05:43:15.158771 3158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:57783
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:43:15.177314 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:36133/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41995
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:43:15.288494 3513 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:43:15.288851 3513 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:43:15.288944 3513 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:43:15.290796 3513 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:43:15.290920 3513 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:43:15.291006 3513 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:43:15.293213 3513 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.213.212:36133/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3513
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:43:15.293710 3513 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:43:15.294065 3513 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:43:15.301159 3521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:43:15.301452 3518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:43:15.301580 3519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:15.302016 3513 server_base.cc:1034] running on GCE node
I20260926 05:43:15.302234 3513 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:43:15.302495 3513 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:43:15.303668 3513 hybrid_clock.cc:648] HybridClock initialized: now 1790401395303636 us; error 45 us; skew 500 ppm
Sep 26 05:43:15 dist-test-slave-1fdt krb5kdc[2682](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:43:15.306917 3513 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:43:15.307433 3513 webserver.cc:460] Webserver started at http://127.23.213.195:42617/ using document root <none> and password file <none>
I20260926 05:43:15.307690 3513 fs_manager.cc:359] Metadata directory not provided
I20260926 05:43:15.307741 3513 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:43:15.307864 3513 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:43:15 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:15.331537 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "04abd562a05246c69ee044695c555b4e"
format_stamp: "Formatted at 2026-09-26 05:43:15 on dist-test-slave-1fdt"
server_key: "218e9c4ab112e44aba0553573c67f144"
server_key_iv: "bc5d8b9427c791af532b0fcd87b75a7d"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:15.332113 3513 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "04abd562a05246c69ee044695c555b4e"
format_stamp: "Formatted at 2026-09-26 05:43:15 on dist-test-slave-1fdt"
server_key: "218e9c4ab112e44aba0553573c67f144"
server_key_iv: "bc5d8b9427c791af532b0fcd87b75a7d"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:15.334312 3513 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 05:43:15.353495 3528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:15.354483 3513 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260926 05:43:15.354598 3513 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "04abd562a05246c69ee044695c555b4e"
format_stamp: "Formatted at 2026-09-26 05:43:15 on dist-test-slave-1fdt"
server_key: "218e9c4ab112e44aba0553573c67f144"
server_key_iv: "bc5d8b9427c791af532b0fcd87b75a7d"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:15.354702 3513 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:43:15.373672 3513 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:43:15.376184 3513 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:43:15.376346 3513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:43:15.376567 3513 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:43:15.376971 3513 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:43:15.377020 3513 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:15.377053 3513 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:43:15.377116 3513 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:15.384151 3513 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:38515
I20260926 05:43:15.384665 3513 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:43:15.384966 3641 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:38515 every 8 connection(s)
I20260926 05:43:15.385952 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 3513
I20260926 05:43:15.386044 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:43:15.386215 24407 external_mini_cluster.cc:1428] Setting key 0ba4b6609b38ce60902f797d164ddb6e
Sep 26 05:43:15 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401395, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:43:15.397428 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.386836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:42055 (local address 127.23.213.254:41995)
0926 05:43:15.386914 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.386916 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.386925 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:43:15.387764 (+ 839us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.387766 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.387777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.387831 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:43:15.388573 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.389371 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.390883 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.391193 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.391233 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.393992 (+ 2759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:43:15.393994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:43:15.393994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:43:15.394002 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:43:15.395907 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:15.396368 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:15.396369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:15.396370 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:15.396425 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:15.396704 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:15.396705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:15.396705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:15.396835 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:43:15.396873 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.397246 (+ 373us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.397329 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 05:43:15.398393 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.386465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41995 (local address 127.23.213.195:42055)
0926 05:43:15.386639 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.386686 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:15.387656 (+ 970us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:15.387927 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:15.387931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:15.388165 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:43:15.388476 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.388483 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.389475 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.389476 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.390783 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.390786 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.391314 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.391315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.391455 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.392116 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:43:15.392126 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:43:15.393887 (+ 1761us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:43:15.395999 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:43:15.396006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:43:15.396008 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:43:15.396285 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:43:15.396492 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:43:15.396493 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:43:15.396493 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:43:15.396627 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:43:15.396915 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:43:15.396921 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:43:15.397080 (+ 159us) client_negotiation.cc:769] Sending connection context
0926 05:43:15.397878 (+ 798us) client_negotiation.cc:240] Negotiation successful
0926 05:43:15.398119 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":83,"threads_started":1}
I20260926 05:43:15.401192 3642 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41995
I20260926 05:43:15.401329 3642 heartbeater.cc:464] Registering TS with master...
I20260926 05:43:15.401551 3642 heartbeater.cc:511] Master 127.23.213.254:41995 requested a full tablet report, sending...
I20260926 05:43:15.402026 3158 ts_manager.cc:194] Registered new tserver with Master: 04abd562a05246c69ee044695c555b4e (127.23.213.195:38515)
I20260926 05:43:15.402523 3158 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:42055
I20260926 05:43:15.409219 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:43:15.419092 3373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.410896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47446 (local address 127.23.213.254:41995)
0926 05:43:15.410943 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.410945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.411230 (+ 285us) server_negotiation.cc:407] Connection header received
0926 05:43:15.411286 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.411287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.411299 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.411362 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:43:15.412096 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.412888 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.414060 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.414368 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.414435 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.415996 (+ 1561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:43:15.415998 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:43:15.415998 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:43:15.416006 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:43:15.417858 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:15.418262 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:15.418263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:15.418263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:15.418316 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:43:15.418497 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:43:15.418498 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:43:15.418498 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:43:15.418635 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:43:15.418712 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.418916 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.418992 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:43:15.421593 3158 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47446:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:43:15.424037 3158 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:43:15.432406 3654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.426858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:38515 (local address 127.0.0.1:52626)
0926 05:43:15.427471 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.427536 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:15.427766 (+ 230us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:15.428000 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:15.428003 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:15.428006 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:43:15.428089 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.428091 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.429412 (+ 1321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.429413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.431032 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.431034 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.432114 (+ 1080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.432115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.432195 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.432208 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:43:15.432271 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 05:43:15.432320 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":585,"thread_start_us":78,"threads_started":1}
I20260926 05:43:15.432868 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.427212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52626 (local address 127.23.213.195:38515)
0926 05:43:15.427405 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.427410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.427661 (+ 251us) server_negotiation.cc:407] Connection header received
0926 05:43:15.427884 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.427887 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.427904 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.427980 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:43:15.428188 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.429315 (+ 1127us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.431151 (+ 1836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.431934 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.431998 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.432053 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.432732 (+ 679us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.432775 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":87,"threads_started":1}
I20260926 05:43:15.432983 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.427117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60866 (local address 127.23.213.194:36837)
0926 05:43:15.428380 (+ 1263us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.428385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.428394 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:43:15.428435 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.428437 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.428452 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.428526 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:43:15.428727 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.430047 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.431365 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.432418 (+ 1053us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.432498 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.432544 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.432870 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.432911 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1231,"thread_start_us":78,"threads_started":1}
I20260926 05:43:15.432798 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.426872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:36837 (local address 127.0.0.1:60866)
0926 05:43:15.428085 (+ 1213us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.428096 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:15.428141 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:15.428601 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:15.428602 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:15.428604 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:43:15.428648 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.428650 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.430147 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.430148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.431235 (+ 1087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.431238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.432577 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.432578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.432643 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.432647 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:43:15.432692 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:43:15.432730 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1196,"thread_start_us":91,"threads_started":1}
I20260926 05:43:15.434201 3575 tablet_service.cc:1467] Processing CreateTablet for tablet de1094ea6bfe4601b66b9006cd5eaa49 (DEFAULT_TABLE table=test-table [id=9580670772254187ab152555090fa495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:43:15.434602 3575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1094ea6bfe4601b66b9006cd5eaa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:43:15.437997 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.427219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47456 (local address 127.23.213.193:43203)
0926 05:43:15.433896 (+ 6677us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.433901 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.433912 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:43:15.433947 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.433949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.433967 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.434048 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:43:15.434236 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.435329 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.436661 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.437376 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.437788 (+ 412us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.437859 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.437885 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.437923 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6640,"thread_start_us":56,"threads_started":1}
I20260926 05:43:15.439945 3660 tablet_bootstrap.cc:492] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Bootstrap starting.
I20260926 05:43:15.437682 3656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.426874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:43203 (local address 127.0.0.1:47456)
0926 05:43:15.428216 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.428224 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:15.428259 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:15.434122 (+ 5863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:15.434124 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:15.434126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:43:15.434165 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.434167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.435429 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.435430 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.436553 (+ 1123us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.436555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.437468 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.437469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.437530 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.437535 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:43:15.437562 (+ 27us) client_negotiation.cc:240] Negotiation successful
0926 05:43:15.437610 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1329,"thread_start_us":85,"threads_started":1}
I20260926 05:43:15.441011 3660 tablet_bootstrap.cc:654] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Neither blocks nor log segments found. Creating new log.
I20260926 05:43:15.440892 3304 tablet_service.cc:1467] Processing CreateTablet for tablet de1094ea6bfe4601b66b9006cd5eaa49 (DEFAULT_TABLE table=test-table [id=9580670772254187ab152555090fa495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:43:15.441247 3304 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1094ea6bfe4601b66b9006cd5eaa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:43:15.441293 3660 log.cc:824] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Log is configured to *not* fsync() on all Append() calls
I20260926 05:43:15.444566 3662 tablet_bootstrap.cc:492] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db: Bootstrap starting.
I20260926 05:43:15.445273 3662 tablet_bootstrap.cc:654] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db: Neither blocks nor log segments found. Creating new log.
I20260926 05:43:15.445570 3662 log.cc:824] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db: Log is configured to *not* fsync() on all Append() calls
I20260926 05:43:15.445995 3442 tablet_service.cc:1467] Processing CreateTablet for tablet de1094ea6bfe4601b66b9006cd5eaa49 (DEFAULT_TABLE table=test-table [id=9580670772254187ab152555090fa495]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:43:15.446369 3442 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de1094ea6bfe4601b66b9006cd5eaa49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:43:15.446693 3660 tablet_bootstrap.cc:492] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: No bootstrap required, opened a new log
I20260926 05:43:15.446775 3660 ts_tablet_manager.cc:1397] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 05:43:15.448828 3660 raft_consensus.cc:348] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.448998 3660 raft_consensus.cc:374] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:43:15.449028 3660 raft_consensus.cc:729] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04abd562a05246c69ee044695c555b4e, State: Initialized, Role: FOLLOWER
I20260926 05:43:15.449168 3660 consensus_queue.cc:260] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.449422 3660 ts_tablet_manager.cc:1428] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:43:15.449537 3642 heartbeater.cc:503] Master 127.23.213.254:41995 was elected leader, sending a full tablet report...
I20260926 05:43:15.451642 3664 tablet_bootstrap.cc:492] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Bootstrap starting.
I20260926 05:43:15.452632 3664 tablet_bootstrap.cc:654] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Neither blocks nor log segments found. Creating new log.
I20260926 05:43:15.452936 3664 log.cc:824] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:43:15.453828 3662 tablet_bootstrap.cc:492] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db: No bootstrap required, opened a new log
I20260926 05:43:15.453908 3662 ts_tablet_manager.cc:1397] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.001s
I20260926 05:43:15.456380 3662 raft_consensus.cc:348] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.456554 3662 raft_consensus.cc:374] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:43:15.456585 3662 raft_consensus.cc:729] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a474cdb8ab614db1a177e1cf8a3525db, State: Initialized, Role: FOLLOWER
I20260926 05:43:15.456734 3662 consensus_queue.cc:260] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.457051 3662 ts_tablet_manager.cc:1428] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 05:43:15.457176 3370 heartbeater.cc:503] Master 127.23.213.254:41995 was elected leader, sending a full tablet report...
I20260926 05:43:15.458243 3664 tablet_bootstrap.cc:492] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: No bootstrap required, opened a new log
I20260926 05:43:15.458318 3664 ts_tablet_manager.cc:1397] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.000s
I20260926 05:43:15.460430 3664 raft_consensus.cc:348] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.460602 3664 raft_consensus.cc:374] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:43:15.460632 3664 raft_consensus.cc:729] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d8d2cb5b6c45a981545b0bed86919a, State: Initialized, Role: FOLLOWER
I20260926 05:43:15.460734 3664 consensus_queue.cc:260] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.460971 3664 ts_tablet_manager.cc:1428] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:43:15.461023 3508 heartbeater.cc:503] Master 127.23.213.254:41995 was elected leader, sending a full tablet report...
Sep 26 05:43:15 dist-test-slave-1fdt krb5kdc[2682](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401394, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:15.596803 3665 raft_consensus.cc:482] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:43:15.596957 3665 raft_consensus.cc:504] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.597378 3665 leader_election.cc:288] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d8d2cb5b6c45a981545b0bed86919a (127.23.213.194:36837), a474cdb8ab614db1a177e1cf8a3525db (127.23.213.193:43203)
I20260926 05:43:15.601636 3644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.597561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:36837 (local address 127.23.213.195:52021)
0926 05:43:15.597623 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.597638 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:15.597685 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:15.598312 (+ 627us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:15.598313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:15.598315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:43:15.598373 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.598375 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.599301 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.599302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.600590 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.600592 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.601364 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.601365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.601437 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.601440 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:43:15.601485 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:43:15.601527 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42,"mutex_wait_us":37}
I20260926 05:43:15.601716 3658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.597849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:58887 (local address 127.23.213.193:43203)
0926 05:43:15.597898 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.597900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.597911 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:43:15.597940 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.597941 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.597950 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.598011 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:43:15.598409 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.599182 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.600732 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.601311 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.601360 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.601368 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.601599 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.601642 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:43:15.601852 3657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.598098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:52021 (local address 127.23.213.194:36837)
0926 05:43:15.598163 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.598165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.598174 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:43:15.598198 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.598199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.598207 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.598260 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:43:15.598441 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.599183 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.600710 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.601260 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.601325 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.601333 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.601735 (+ 402us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.601793 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:43:15.602574 3669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.597561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:43203 (local address 127.23.213.195:58887)
0926 05:43:15.597788 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.597800 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:43:15.597872 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:43:15.598225 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:43:15.598226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:43:15.598228 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:43:15.598312 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.598315 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.599278 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.599279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.600635 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:43:15.600638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.601431 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.601432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:43:15.601513 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.601517 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:43:15.602456 (+ 939us) client_negotiation.cc:240] Negotiation successful
0926 05:43:15.602498 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":17,"thread_start_us":127,"threads_started":1}
I20260926 05:43:15.602592 3462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1094ea6bfe4601b66b9006cd5eaa49" candidate_uuid: "04abd562a05246c69ee044695c555b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" is_pre_election: true
I20260926 05:43:15.602751 3462 raft_consensus.cc:2454] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04abd562a05246c69ee044695c555b4e in term 0.
I20260926 05:43:15.603012 3532 leader_election.cc:302] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd562a05246c69ee044695c555b4e, d9d8d2cb5b6c45a981545b0bed86919a; no voters:
I20260926 05:43:15.603144 3665 raft_consensus.cc:2781] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:43:15.603183 3665 raft_consensus.cc:482] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:43:15.603202 3665 raft_consensus.cc:3037] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:43:15.603199 3324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1094ea6bfe4601b66b9006cd5eaa49" candidate_uuid: "04abd562a05246c69ee044695c555b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a474cdb8ab614db1a177e1cf8a3525db" is_pre_election: true
I20260926 05:43:15.603355 3324 raft_consensus.cc:2454] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 04abd562a05246c69ee044695c555b4e in term 0.
I20260926 05:43:15.604072 3665 raft_consensus.cc:504] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.604210 3665 leader_election.cc:288] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [CANDIDATE]: Term 1 election: Requested vote from peers d9d8d2cb5b6c45a981545b0bed86919a (127.23.213.194:36837), a474cdb8ab614db1a177e1cf8a3525db (127.23.213.193:43203)
I20260926 05:43:15.604348 3324 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1094ea6bfe4601b66b9006cd5eaa49" candidate_uuid: "04abd562a05246c69ee044695c555b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a474cdb8ab614db1a177e1cf8a3525db"
I20260926 05:43:15.604404 3462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de1094ea6bfe4601b66b9006cd5eaa49" candidate_uuid: "04abd562a05246c69ee044695c555b4e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d8d2cb5b6c45a981545b0bed86919a"
I20260926 05:43:15.604423 3324 raft_consensus.cc:3037] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:43:15.604473 3462 raft_consensus.cc:3037] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:43:15.605268 3462 raft_consensus.cc:2454] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04abd562a05246c69ee044695c555b4e in term 1.
I20260926 05:43:15.605342 3324 raft_consensus.cc:2454] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 04abd562a05246c69ee044695c555b4e in term 1.
I20260926 05:43:15.605430 3532 leader_election.cc:302] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 04abd562a05246c69ee044695c555b4e, d9d8d2cb5b6c45a981545b0bed86919a; no voters:
I20260926 05:43:15.605509 3665 raft_consensus.cc:2781] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:43:15.605665 3665 raft_consensus.cc:686] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 1 LEADER]: Becoming Leader. State: Replica: 04abd562a05246c69ee044695c555b4e, State: Running, Role: LEADER
I20260926 05:43:15.605784 3665 consensus_queue.cc:237] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } }
I20260926 05:43:15.606642 3158 catalog_manager.cc:5495] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e reported cstate change: term changed from 0 to 1, leader changed from <none> to 04abd562a05246c69ee044695c555b4e (127.23.213.195). New cstate: current_term: 1 leader_uuid: "04abd562a05246c69ee044695c555b4e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "04abd562a05246c69ee044695c555b4e" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 38515 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 } health_report { overall_health: UNKNOWN } } }
I20260926 05:43:15.624312 3659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.620919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.23.213.195:38515)
0926 05:43:15.621030 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:15.621032 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:15.621039 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:43:15.621077 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:15.621078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:15.621100 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:43:15.621170 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:43:15.621372 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.622144 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.623285 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:43:15.623598 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:43:15.623673 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:43:15.623874 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:43:15.624032 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:43:15.624134 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 05:43:15.624207 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 05:43:15.624230 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 05:43:15.626672 3462 raft_consensus.cc:1261] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 1 FOLLOWER]: Refusing update from remote peer 04abd562a05246c69ee044695c555b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:43:15.626668 3324 raft_consensus.cc:1261] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db [term 1 FOLLOWER]: Refusing update from remote peer 04abd562a05246c69ee044695c555b4e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:43:15.627018 3665 consensus_queue.cc:1035] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d8d2cb5b6c45a981545b0bed86919a" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 36837 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:43:15.627200 3670 consensus_queue.cc:1035] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [LEADER]: Connected to new peer: Peer: permanent_uuid: "a474cdb8ab614db1a177e1cf8a3525db" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 43203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:43:15.628880 3371 tablet.cc:2369] T de1094ea6bfe4601b66b9006cd5eaa49 P a474cdb8ab614db1a177e1cf8a3525db: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:43:15.630858 3673 mvcc.cc:205] Tried to move back new op lower bound from 7333484116483555328 to 7333484116401852416. Current Snapshot: MvccSnapshot[applied={T|T < 7333484116483555328}]
I20260926 05:43:15.631414 3676 mvcc.cc:205] Tried to move back new op lower bound from 7333484116483555328 to 7333484116401852416. Current Snapshot: MvccSnapshot[applied={T|T < 7333484116483555328}]
I20260926 05:43:15.633503 3674 mvcc.cc:205] Tried to move back new op lower bound from 7333484116483555328 to 7333484116401852416. Current Snapshot: MvccSnapshot[applied={T|T < 7333484116483555328}]
I20260926 05:43:15.636129 3158 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:43:15.636240 3158 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47446:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:43:15.637012 3158 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4648d28ba55b49d5baac032b2f7362a6: Sending DeleteTablet for 3 replicas of tablet de1094ea6bfe4601b66b9006cd5eaa49
I20260926 05:43:15.637336 3304 tablet_service.cc:1514] Processing DeleteTablet for tablet de1094ea6bfe4601b66b9006cd5eaa49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:43:15 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:47456
I20260926 05:43:15.637377 3442 tablet_service.cc:1514] Processing DeleteTablet for tablet de1094ea6bfe4601b66b9006cd5eaa49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:43:15 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:60866
I20260926 05:43:15.637844 3575 tablet_service.cc:1514] Processing DeleteTablet for tablet de1094ea6bfe4601b66b9006cd5eaa49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:43:15 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:52626
I20260926 05:43:15.638206 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 3239
I20260926 05:43:15.645921 3684 tablet_replica.cc:330] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: stopping tablet replica
I20260926 05:43:15.646130 3684 raft_consensus.cc:2229] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:43:15.646390 3684 raft_consensus.cc:2258] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:43:15.646553 3683 tablet_replica.cc:330] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: stopping tablet replica
I20260926 05:43:15.646699 3683 raft_consensus.cc:2229] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:43:15.646925 3683 raft_consensus.cc:2258] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:43:15.647298 3684 ts_tablet_manager.cc:1905] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:43:15.647468 3683 ts_tablet_manager.cc:1905] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:43:15.651712 3684 ts_tablet_manager.cc:1918] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:43:15.651875 3684 log.cc:1196] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/de1094ea6bfe4601b66b9006cd5eaa49
I20260926 05:43:15.652228 3684 ts_tablet_manager.cc:1939] T de1094ea6bfe4601b66b9006cd5eaa49 P 04abd562a05246c69ee044695c555b4e: Deleting consensus metadata
I20260926 05:43:15.653230 3683 ts_tablet_manager.cc:1918] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:43:15.653376 3683 log.cc:1196] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/de1094ea6bfe4601b66b9006cd5eaa49
I20260926 05:43:15.653692 3683 ts_tablet_manager.cc:1939] T de1094ea6bfe4601b66b9006cd5eaa49 P d9d8d2cb5b6c45a981545b0bed86919a: Deleting consensus metadata
I20260926 05:43:15.654211 3145 catalog_manager.cc:4841] TS d9d8d2cb5b6c45a981545b0bed86919a (127.23.213.194:36837): tablet de1094ea6bfe4601b66b9006cd5eaa49 (table test-table [id=9580670772254187ab152555090fa495]) successfully deleted
I20260926 05:43:15.654331 3144 catalog_manager.cc:4841] TS 04abd562a05246c69ee044695c555b4e (127.23.213.195:38515): tablet de1094ea6bfe4601b66b9006cd5eaa49 (table test-table [id=9580670772254187ab152555090fa495]) successfully deleted
W20260926 05:43:15.654594 3142 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:43203 (error 108)
I20260926 05:43:15.655578 3655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:15.655451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:43203 (local address 127.0.0.1:47458)
0926 05:43:15.655498 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:43:15.655539 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:43203: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 05:43:15.655603 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 3378
W20260926 05:43:15.657181 3142 catalog_manager.cc:4584] TS a474cdb8ab614db1a177e1cf8a3525db (127.23.213.193:43203): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet de1094ea6bfe4601b66b9006cd5eaa49: Network error: Client connection negotiation failed: client connection to 127.23.213.193:43203: connect: Connection refused (error 111)
I20260926 05:43:15.662005 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 3513
I20260926 05:43:15.668682 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 3126
I20260926 05:43:15.675350 24407 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 05:43:16.019942 24407 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 05:43:16.020077 24407 mini_ranger.cc:66] Stopping Ranger...
I20260926 05:43:16.367383 24407 mini_ranger.cc:68] Stopped Ranger
2026-09-26 05:43:16.367 UTC [2712] LOG: received smart shutdown request
2026-09-26 05:43:16.370 UTC [2712] LOG: background worker "logical replication launcher" (PID 2721) exited with exit code 1
2026-09-26 05:43:16.371 UTC [2716] LOG: shutting down
2026-09-26 05:43:16.529 UTC [2712] LOG: database system is shut down
2026-09-26T05:43:16Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (46792 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:43:16 dist-test-slave-1fdt krb5kdc[3694](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:43:16 dist-test-slave-1fdt krb5kdc[3694](info): set up 2 sockets
Sep 26 05:43:16 dist-test-slave-1fdt krb5kdc[3694](info): commencing operation
krb5kdc: starting...
W20260926 05:43:17.681846 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:43:17 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401397, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:43:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:43:17Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/rangeradmin_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/rangeradmin_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/rangerlookup_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/rangerlookup_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
I20260926 05:43:17.846666 24407 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPYhTSt/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/postgres -l logfile start
2026-09-26 05:43:20.381 UTC [3724] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 05:43:20.381 UTC [3724] LOG: listening on IPv4 address "127.23.213.212", port 32883
2026-09-26 05:43:20.384 UTC [3724] LOG: listening on Unix socket "/tmp/.s.PGSQL.32883"
2026-09-26 05:43:20.405 UTC [3727] LOG: database system was shut down at 2026-09-26 05:43:18 UTC
2026-09-26 05:43:20.409 UTC [3724] LOG: database system is ready to accept connections
I20260926 05:43:21.361536 24407 mini_postgres.cc:96] Postgres bound to 32883
2026-09-26 05:43:21.367 UTC [3735] FATAL: database "slave" does not exist
127.23.213.212:32883 - accepting connections
I20260926 05:43:21.368105 24407 mini_ranger.cc:161] Starting Ranger...
I20260926 05:43:21.382822 24407 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 05:43:21.864370 24407 mini_ranger.cc:87] Created ranger Postgres database
I20260926 05:43:21.864455 24407 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-admin
2026-09-26 05:43:22,144 [I] DB FLAVOR :POSTGRES
2026-09-26 05:43:22,145 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:43:22,145 [I] Checking connection..
2026-09-26 05:43:22,145 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 05:43:22,406 [I] Checking connection passed.
2026-09-26 05:43:22,406 [I] --------- Verifying version history table ---------
2026-09-26 05:43:22,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:43:22,631 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:43:22,632 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:43:22,884 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 05:43:22,884 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 05:43:22,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 05:43:23.133 UTC [3799] WARNING: there is no transaction in progress
2026-09-26 05:43:23,139 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 05:43:23,140 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 05:43:23,398 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 05:43:23,398 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:43:23,398 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 05:43:23,624 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 05:43:23,880 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-1fdt.c.gcp-upstream.internal','N') ;"
2026-09-26 05:43:24,109 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:43:24,358 [I] Table x_portal_user does not exist in database ranger
2026-09-26 05:43:24,358 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:43:24,608 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 05:43:24,608 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:43:24,841 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:43:25,079 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 05:43:25,079 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 05:43:26.072 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.090 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.107 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.124 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.143 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.160 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.227 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.234 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.248 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.259 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.270 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.284 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.293 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.300 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.307 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26.314 UTC [3907] WARNING: there is no transaction in progress
2026-09-26 05:43:26,952 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 05:43:26,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 05:43:27,186 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 05:43:27,186 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 05:43:27,426 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 05:43:27,426 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 05:43:27,683 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 05:43:27,683 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 05:43:27,929 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 05:43:28,174 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-1fdt.c.gcp-upstream.internal';"
2026-09-26 05:43:28,437 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 05:43:28,438 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 05:43:28,670 [I] DB_PATCHES have already been applied
I20260926 05:43:28.675819 24407 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPYhTSt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 05:43:28.675884 24407 mini_ranger.cc:193] Using host: 127.23.213.212
I20260926 05:43:28.678956 24407 mini_ranger.cc:227] Ranger admin URL: http://127.23.213.212:59555
Sep 26, 2026 5:43:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:43:28 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:43:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 5:43:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 5:43:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 5:43:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 5:43:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 05:43:29.788300 24407 mini_ranger.cc:160] Time spent starting Ranger: real 8.420s user 0.001s sys 0.007s
Sep 26, 2026 5:43:49 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 5:43:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.XKeyREST
Sep 26, 2026 5:43:49 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 5:43:49 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 5:43:50 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 05:43:51.067996 24407 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.23.213.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.23.213.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/rangerkms_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.23.213.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/rangerkms_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.23.213.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.23.213.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.23.213.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/HTTP_127.23.213.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 05:43:51 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401431, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 05:43:51.152681 24407 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 05:43:51.161886 24407 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 05:43:52.079126 24407 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 05:43:52.079196 24407 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms
I20260926 05:43:52.079209 24407 mini_ranger_kms.cc:227] Using postgres at 127.23.213.212:32883
2026-09-26 05:43:52,248 [I] DB FLAVOR :POSTGRES
2026-09-26 05:43:52,249 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 05:43:52,249 [I] Checking connection
2026-09-26 05:43:52,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 05:43:52,476 [I] connection success
2026-09-26 05:43:52,476 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 05:43:52,476 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 05:43:52,476 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 05:43:52,740 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 05:43:52,740 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 05:43:52,740 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 05:43:52,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.23.213.212:32883/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 05:43:53,008 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 05:43:53.425529 24407 mini_ranger_kms.cc:310] Created kms service
I20260926 05:43:53.622949 24407 mini_ranger_kms.cc:326] Created kudu user
I20260926 05:43:53.664309 24407 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 05:43:53 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401431, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:54.061626 24407 mini_ranger_kms.cc:384] Added ranger policy
I20260926 05:43:54.061769 24407 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPYhTSt/build/release/bin/postgresql.jar:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPYhTSt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 05:43:54.061792 24407 mini_ranger_kms.cc:240] Using host: 127.23.213.212
I20260926 05:43:54.065431 24407 mini_ranger_kms.cc:276] Ranger KMS PID: 4087
I20260926 05:43:54.065517 24407 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.23.213.212:49655
Sep 26, 2026 5:43:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 5:43:55 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 5:43:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 5:43:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 5:43:55 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 5:43:55 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 05:43:56.357391 24407 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.205s user 0.001s sys 0.151s
I20260926 05:43:56.357519 24407 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 05:43:56.357570 24407 mini_ranger_kms.cc:401] 127.23.213.212:49655/kms/v1/keys
Sep 26 05:43:56 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401436, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.213.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 05:43:57 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401436, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.213.212@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
Sep 26 05:43:57 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401436, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.23.213.212@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
Sep 26 05:43:59 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:43:59.768514 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40509
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:40509
--ranger_config_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:43:59.862429 4138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:43:59.862670 4138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:43:59.862708 4138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:43:59.863903 4138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:43:59.863970 4138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:43:59.863996 4138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:43:59.864038 4138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:43:59.864071 4138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:43:59.865598 4138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
--ranger_kms_url=127.23.213.212:49655/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:40509
--ranger_config_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:43:59.865893 4138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:43:59.866147 4138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:43:59.871208 4144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:43:59.871227 4143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:59.871299 4138 server_base.cc:1034] running on GCE node
W20260926 05:43:59.871430 4146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:43:59.871788 4138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:43:59.872097 4138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:43:59.873265 4138 hybrid_clock.cc:648] HybridClock initialized: now 1790401439873235 us; error 47 us; skew 500 ppm
Sep 26 05:43:59 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401439, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:43:59.876017 4138 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:43:59.876466 4138 webserver.cc:460] Webserver started at http://127.23.213.254:39775/ using document root <none> and password file <none>
I20260926 05:43:59.876708 4138 fs_manager.cc:359] Metadata directory not provided
I20260926 05:43:59.876772 4138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:43:59.876932 4138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:43:59 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401439, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:43:59.921917 4138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "421f41da4efe4d87bfbe161e328ce852"
format_stamp: "Formatted at 2026-09-26 05:43:59 on dist-test-slave-1fdt"
server_key: "38520c2aa9622feb9a8a9d2144b54773"
server_key_iv: "aa92fa4c8e7bbf403c077196b2c73864"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:59.922355 4138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "421f41da4efe4d87bfbe161e328ce852"
format_stamp: "Formatted at 2026-09-26 05:43:59 on dist-test-slave-1fdt"
server_key: "38520c2aa9622feb9a8a9d2144b54773"
server_key_iv: "aa92fa4c8e7bbf403c077196b2c73864"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:59.923853 4138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:43:59.935614 4153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:43:59.935882 4138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:43:59.935992 4138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "421f41da4efe4d87bfbe161e328ce852"
format_stamp: "Formatted at 2026-09-26 05:43:59 on dist-test-slave-1fdt"
server_key: "38520c2aa9622feb9a8a9d2144b54773"
server_key_iv: "aa92fa4c8e7bbf403c077196b2c73864"
server_key_version: "kuduclusterkey@0"
I20260926 05:43:59.936117 4138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:43:59.979176 4138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:43:59.981057 4138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:43:59.981225 4138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:43:59.986209 4138 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:40509
I20260926 05:43:59.986287 4205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:40509 every 8 connection(s)
I20260926 05:43:59.986657 4138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:43:59.987350 4206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:43:59.989564 4206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Bootstrap starting.
I20260926 05:43:59.990177 4206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Neither blocks nor log segments found. Creating new log.
I20260926 05:43:59.990468 4206 log.cc:824] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Log is configured to *not* fsync() on all Append() calls
I20260926 05:43:59.991195 4206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: No bootstrap required, opened a new log
I20260926 05:43:59.993160 4206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:43:59.993312 4206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:43:59.993336 4206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421f41da4efe4d87bfbe161e328ce852, State: Initialized, Role: FOLLOWER
I20260926 05:43:59.993505 4206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:43:59.993580 4206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:43:59.993602 4206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:43:59.993670 4206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:43:59.994266 4206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:43:59.994352 4206 leader_election.cc:302] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 421f41da4efe4d87bfbe161e328ce852; no voters:
I20260926 05:43:59.994557 4206 leader_election.cc:288] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:43:59.994588 4209 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:43:59.994702 4209 raft_consensus.cc:686] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 LEADER]: Becoming Leader. State: Replica: 421f41da4efe4d87bfbe161e328ce852, State: Running, Role: LEADER
I20260926 05:43:59.994781 4209 consensus_queue.cc:237] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:43:59.994812 4206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:43:59.995097 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "421f41da4efe4d87bfbe161e328ce852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } } }
I20260926 05:43:59.995136 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 4138
I20260926 05:43:59.995199 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: This master's current role is: LEADER
I20260926 05:43:59.995231 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:43:59.995343 4209 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 421f41da4efe4d87bfbe161e328ce852. Latest consensus state: current_term: 1 leader_uuid: "421f41da4efe4d87bfbe161e328ce852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } } }
I20260926 05:43:59.995398 24407 external_mini_cluster.cc:1428] Setting key 12782600834805c1b0a0b70b6e9f6d59
I20260926 05:43:59.995434 4209 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: This master's current role is: LEADER
I20260926 05:43:59.995600 4214 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:43:59.996131 4214 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:43:59.996388 4206 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 05:44:00.002517 4214 catalog_manager.cc:1347] Generated new cluster ID: 6ae50806cd014660b0459cb94c161b57
I20260926 05:44:00.002666 4214 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:44:00 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:00.012158 4226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:43:59.996031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56984 (local address 127.23.213.254:40509)
0926 05:43:59.999350 (+ 3319us) server_negotiation.cc:206] Beginning negotiation
0926 05:43:59.999387 (+ 37us) server_negotiation.cc:399] Waiting for connection header
0926 05:43:59.999454 (+ 67us) server_negotiation.cc:407] Connection header received
0926 05:43:59.999749 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:43:59.999758 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:43:59.999852 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:00.000077 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:00.001193 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:00.003015 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:00.004120 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:00.004777 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:00.004893 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:00.007675 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:00.007679 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:00.007679 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:00.007691 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:00.010051 (+ 2360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:00.010417 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:00.010420 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:00.010421 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:00.010532 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:00.010728 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:00.010729 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:00.010729 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:00.011073 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:00.011191 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:00.011754 (+ 563us) server_negotiation.cc:299] Negotiation successful
0926 05:44:00.011876 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3192,"thread_start_us":82,"threads_started":1}
I20260926 05:44:00.025995 4214 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:44:00.026821 4214 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:00.038885 4214 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Generated new TSK 0
I20260926 05:44:00.039214 4214 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:44:00.823026 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40509
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:00.927678 4251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:00.927920 4251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:00.927956 4251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:00.929291 4251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:00.929349 4251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:00.929388 4251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:44:00.930941 4251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.23.213.212:49655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:00.931186 4251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:00.931452 4251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:00.937631 4259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:00.937788 4251 server_base.cc:1034] running on GCE node
W20260926 05:44:00.937836 4256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:00.938081 4257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:00.938308 4251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:00.938604 4251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:00.939816 4251 hybrid_clock.cc:648] HybridClock initialized: now 1790401440939795 us; error 33 us; skew 500 ppm
Sep 26 05:44:00 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401440, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:00.943069 4251 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:44:00.943583 4251 webserver.cc:460] Webserver started at http://127.23.213.193:44491/ using document root <none> and password file <none>
I20260926 05:44:00.943897 4251 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:00.943969 4251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:00.944129 4251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:44:00 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401440, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:00.966239 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "14de182e1a724ffb83d3fd2d7231b563"
format_stamp: "Formatted at 2026-09-26 05:44:00 on dist-test-slave-1fdt"
server_key: "276aca745d7cfb9cb448984afe11da3d"
server_key_iv: "1f2fb6ccf5c7b6aed8e5898556a89a95"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:00.966696 4251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "14de182e1a724ffb83d3fd2d7231b563"
format_stamp: "Formatted at 2026-09-26 05:44:00 on dist-test-slave-1fdt"
server_key: "276aca745d7cfb9cb448984afe11da3d"
server_key_iv: "1f2fb6ccf5c7b6aed8e5898556a89a95"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:00.969533 4251 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 05:44:00.986040 4268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:00.986366 4251 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:00.986466 4251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "14de182e1a724ffb83d3fd2d7231b563"
format_stamp: "Formatted at 2026-09-26 05:44:00 on dist-test-slave-1fdt"
server_key: "276aca745d7cfb9cb448984afe11da3d"
server_key_iv: "1f2fb6ccf5c7b6aed8e5898556a89a95"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:00.986562 4251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:01.019160 4251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:01.021662 4251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:01.021874 4251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:01.022132 4251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:01.022508 4251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:01.022590 4251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.022720 4251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:01.022799 4251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.029883 4251 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:44853
I20260926 05:44:01.030045 4381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:44853 every 8 connection(s)
I20260926 05:44:01.030455 4251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401440, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:01.040504 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 4251
I20260926 05:44:01.040611 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:44:01.040791 24407 external_mini_cluster.cc:1428] Setting key 0d40e05e7756d1b69e62b260d43bf017
I20260926 05:44:01.043313 4384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:01.031376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40509 (local address 127.23.213.193:54287)
0926 05:44:01.031577 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:01.031631 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:01.032276 (+ 645us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:01.032634 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:01.032640 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:01.032896 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:01.033188 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:01.033194 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.034271 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.034272 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:01.035555 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:01.035558 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.036137 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.036138 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:01.036278 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:01.036894 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:01.036905 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:01.038661 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:01.041494 (+ 2833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:01.041502 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:01.041505 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:01.041882 (+ 377us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:01.042108 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:01.042109 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:01.042110 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:01.042255 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:01.042541 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:01.042546 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:01.042729 (+ 183us) client_negotiation.cc:769] Sending connection context
0926 05:44:01.042835 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 05:44:01.043034 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":91,"threads_started":1}
I20260926 05:44:01.043862 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:01.031692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:54287 (local address 127.23.213.254:40509)
0926 05:44:01.031843 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:01.031845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:01.031857 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:44:01.032423 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:01.032424 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:01.032441 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:01.032535 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:01.033304 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.034161 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.035655 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.035991 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.036054 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:01.038772 (+ 2718us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:01.038773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:01.038774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:01.038782 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:01.041397 (+ 2615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:01.041974 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:01.041975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:01.041976 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:01.042042 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:01.042317 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:01.042318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:01.042318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:01.042460 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:01.042516 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:01.043674 (+ 1158us) server_negotiation.cc:299] Negotiation successful
0926 05:44:01.043763 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":83,"threads_started":1}
I20260926 05:44:01.044678 4382 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40509
I20260926 05:44:01.044788 4382 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:01.045007 4382 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:01.045610 4170 ts_manager.cc:194] Registered new tserver with Master: 14de182e1a724ffb83d3fd2d7231b563 (127.23.213.193:44853)
I20260926 05:44:01.046533 4170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:54287
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:44:01.084468 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40509
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 05:44:01.206099 4390 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:01.206360 4390 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:01.206396 4390 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:01.208067 4390 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:01.208194 4390 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:01.208287 4390 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:44:01.210444 4390 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.23.213.212:49655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4390
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:01.210987 4390 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:01.211256 4390 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:01.217712 4397 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.217798 4396 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.217864 4399 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:01.218801 4390 server_base.cc:1034] running on GCE node
I20260926 05:44:01.219028 4390 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:01.219307 4390 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:01.220484 4390 hybrid_clock.cc:648] HybridClock initialized: now 1790401441220453 us; error 44 us; skew 500 ppm
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:01.223717 4390 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:44:01.224263 4390 webserver.cc:460] Webserver started at http://127.23.213.194:38657/ using document root <none> and password file <none>
I20260926 05:44:01.224578 4390 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:01.224644 4390 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:01.224817 4390 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:01.243464 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "b1d43f362b504eb489f8eb3c5685a354"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "b4dbae366e6abf90cc2e8bfff38cd28a"
server_key_iv: "869e2547ba9dcb1e8c367c16ebc4c194"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:01.243952 4390 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "b1d43f362b504eb489f8eb3c5685a354"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "b4dbae366e6abf90cc2e8bfff38cd28a"
server_key_iv: "869e2547ba9dcb1e8c367c16ebc4c194"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:01.245883 4390 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.004s sys 0.000s
I20260926 05:44:01.266290 4406 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.266589 4390 fs_manager.cc:725] Time spent opening block manager: real 0.008s user 0.001s sys 0.000s
I20260926 05:44:01.266705 4390 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "b1d43f362b504eb489f8eb3c5685a354"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "b4dbae366e6abf90cc2e8bfff38cd28a"
server_key_iv: "869e2547ba9dcb1e8c367c16ebc4c194"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:01.266824 4390 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:01.297680 4390 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:01.300294 4390 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:01.300523 4390 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:01.300845 4390 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:01.301266 4390 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:01.301366 4390 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.301445 4390 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:01.301520 4390 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.308094 4390 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:41507
I20260926 05:44:01.308614 4390 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:44:01.311887 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 4390
I20260926 05:44:01.312011 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:44:01.312198 24407 external_mini_cluster.cc:1428] Setting key 9ef1841c444095bae604a1d5d9a6f8a0
I20260926 05:44:01.318274 4519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:41507 every 8 connection(s)
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:01.344336 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:01.320836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:55945 (local address 127.23.213.254:40509)
0926 05:44:01.320888 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:01.320890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:01.322236 (+ 1346us) server_negotiation.cc:407] Connection header received
0926 05:44:01.322921 (+ 685us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:01.322922 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:01.322939 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:01.322995 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:01.333044 (+ 10049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.333934 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.337505 (+ 3571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.337850 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.337902 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:01.341006 (+ 3104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:01.341007 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:01.341008 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:01.341016 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:01.342659 (+ 1643us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:01.343204 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:01.343206 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:01.343206 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:01.343268 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:01.343550 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:01.343551 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:01.343551 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:01.343703 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:01.343748 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:01.344136 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 05:44:01.344234 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:44:01.345111 4523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:01.320534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40509 (local address 127.23.213.194:55945)
0926 05:44:01.322097 (+ 1563us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:01.322151 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:01.322813 (+ 662us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:01.332166 (+ 9353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:01.332173 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:01.332431 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:01.332759 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:01.332767 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.334074 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.334075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:01.337385 (+ 3310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:01.337389 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.338017 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.338018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:01.338165 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:01.338950 (+ 785us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:01.338961 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:01.340887 (+ 1926us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:01.342760 (+ 1873us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:01.342767 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:01.342770 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:01.343080 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:01.343336 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:01.343336 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:01.343337 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:01.343474 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:01.343793 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:01.343799 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:01.343972 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 05:44:01.344699 (+ 727us) client_negotiation.cc:240] Negotiation successful
0926 05:44:01.344871 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1515,"thread_start_us":98,"threads_started":1}
I20260926 05:44:01.346261 4520 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40509
I20260926 05:44:01.346365 4520 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:01.346561 4520 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:01.347010 4170 ts_manager.cc:194] Registered new tserver with Master: b1d43f362b504eb489f8eb3c5685a354 (127.23.213.194:41507)
I20260926 05:44:01.347502 4170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:55945
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:44:01.366756 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40509
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:01.486999 4525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:01.487342 4525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:01.487442 4525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:01.489181 4525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:01.489290 4525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:01.489369 4525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:44:01.491551 4525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.23.213.212:49655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:01.492115 4525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:01.492430 4525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:01.498998 4531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.498932 4530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.499310 4533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:01.500378 4525 server_base.cc:1034] running on GCE node
I20260926 05:44:01.500687 4525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:01.500957 4525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:01.502094 4525 hybrid_clock.cc:648] HybridClock initialized: now 1790401441502070 us; error 39 us; skew 500 ppm
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:01.505103 4525 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:44:01.505606 4525 webserver.cc:460] Webserver started at http://127.23.213.195:33305/ using document root <none> and password file <none>
I20260926 05:44:01.505890 4525 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:01.505940 4525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:01.506072 4525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:01.521543 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "e8aaeb81322b4939804343ed608cbf5d"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "83d3fbc039bcef18df76b7e475f3e59d"
server_key_iv: "089923c3917e8f335d77d2761c999f97"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:01.522116 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "e8aaeb81322b4939804343ed608cbf5d"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "83d3fbc039bcef18df76b7e475f3e59d"
server_key_iv: "089923c3917e8f335d77d2761c999f97"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:01.523932 4525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:01.540766 4540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.541038 4525 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260926 05:44:01.541126 4525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "e8aaeb81322b4939804343ed608cbf5d"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "83d3fbc039bcef18df76b7e475f3e59d"
server_key_iv: "089923c3917e8f335d77d2761c999f97"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:01.541216 4525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:01.561558 4525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:01.564221 4525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:01.564414 4525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:01.564656 4525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:01.565045 4525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:01.565136 4525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.565207 4525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:01.565263 4525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.572016 4525 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:44407
I20260926 05:44:01.572525 4525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:44:01.576066 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 4525
I20260926 05:44:01.576181 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:44:01.576372 24407 external_mini_cluster.cc:1428] Setting key a9f9d1ea1396c532f55c9dce5fd9cfb7
I20260926 05:44:01.574831 4653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:44407 every 8 connection(s)
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:01.588357 4385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:01.575804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:38395 (local address 127.23.213.254:40509)
0926 05:44:01.575852 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:01.575854 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:01.577936 (+ 2082us) server_negotiation.cc:407] Connection header received
0926 05:44:01.578630 (+ 694us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:01.578632 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:01.578644 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:01.578673 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:01.579503 (+ 830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.580238 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.581694 (+ 1456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.582002 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.582049 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:01.584640 (+ 2591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:01.584642 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:01.584642 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:01.584649 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:01.586295 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:01.586743 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:01.586744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:01.586744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:01.586800 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:01.587065 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:01.587065 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:01.587066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:01.587190 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:01.587228 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:01.588178 (+ 950us) server_negotiation.cc:299] Negotiation successful
0926 05:44:01.588264 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:44:01.589104 4656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:01.575497 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40509 (local address 127.23.213.195:38395)
0926 05:44:01.577815 (+ 2318us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:01.577858 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:01.578508 (+ 650us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:01.578860 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:01.578865 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:01.579111 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:01.579390 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:01.579396 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.580355 (+ 959us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.580356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:01.581589 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:01.581592 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:01.582158 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:01.582158 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:01.582290 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:01.582877 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:01.582888 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:01.584536 (+ 1648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:01.586387 (+ 1851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:01.586393 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:01.586396 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:01.586661 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:01.586865 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:01.586866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:01.586866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:01.586996 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:01.587271 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:01.587276 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:01.588011 (+ 735us) client_negotiation.cc:769] Sending connection context
0926 05:44:01.588704 (+ 693us) client_negotiation.cc:240] Negotiation successful
0926 05:44:01.588865 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2273,"thread_start_us":89,"threads_started":1}
I20260926 05:44:01.589859 4654 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40509
I20260926 05:44:01.589960 4654 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:01.590135 4654 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:01.590523 4170 ts_manager.cc:194] Registered new tserver with Master: e8aaeb81322b4939804343ed608cbf5d (127.23.213.195:44407)
I20260926 05:44:01.590988 4170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:38395
I20260926 05:44:01.598479 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:44:01.599141 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 4251
I20260926 05:44:01.606371 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 4390
I20260926 05:44:01.627583 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 4525
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:01.650486 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 4138
I20260926 05:44:01.658294 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40509
--webserver_interface=127.23.213.254
--webserver_port=39775
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:40509
--ranger_config_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:01.762240 4657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:01.762485 4657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:01.762509 4657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:01.763634 4657 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:01.763679 4657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:01.763696 4657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:01.763710 4657 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:01.763725 4657 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:01.765249 4657 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:40509
--ranger_config_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=39775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:01.765470 4657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:01.765640 4657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:01.770771 4665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.770860 4662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.770795 4663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:01.771070 4657 server_base.cc:1034] running on GCE node
I20260926 05:44:01.771355 4657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:01.771658 4657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:01.772835 4657 hybrid_clock.cc:648] HybridClock initialized: now 1790401441772825 us; error 46 us; skew 500 ppm
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:01.775421 4657 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:01.775851 4657 webserver.cc:460] Webserver started at http://127.23.213.254:39775/ using document root <none> and password file <none>
I20260926 05:44:01.776072 4657 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:01.776142 4657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:01.829380 4657 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:01.830425 4673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:01.830699 4657 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:01.830814 4657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "421f41da4efe4d87bfbe161e328ce852"
format_stamp: "Formatted at 2026-09-26 05:43:59 on dist-test-slave-1fdt"
server_key: "38520c2aa9622feb9a8a9d2144b54773"
server_key_iv: "aa92fa4c8e7bbf403c077196b2c73864"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:01.831346 4657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:01.863410 4657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:01.865353 4657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:01.865470 4657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:01.869679 4657 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:40509
I20260926 05:44:01.869809 4725 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:40509 every 8 connection(s)
I20260926 05:44:01.870155 4657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:01.871409 4726 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:44:01.872088 4726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Bootstrap starting.
I20260926 05:44:01.878234 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 4657
I20260926 05:44:01.878577 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:44853
--local_ip_for_outbound_sockets=127.23.213.193
--tserver_master_addrs=127.23.213.254:40509
--webserver_port=44491
--webserver_interface=127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:01.880107 4726 log.cc:824] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:01.881968 4726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:44:01.882254 4726 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Bootstrap complete.
I20260926 05:44:01.884634 4726 raft_consensus.cc:348] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:44:01.884871 4726 raft_consensus.cc:729] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 421f41da4efe4d87bfbe161e328ce852, State: Initialized, Role: FOLLOWER
I20260926 05:44:01.885042 4726 consensus_queue.cc:260] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:44:01.885113 4726 raft_consensus.cc:388] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:44:01.885187 4726 raft_consensus.cc:482] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:44:01.885231 4726 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:44:01.886330 4726 raft_consensus.cc:504] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:44:01.886417 4726 leader_election.cc:302] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 421f41da4efe4d87bfbe161e328ce852; no voters:
I20260926 05:44:01.886653 4726 leader_election.cc:288] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:44:01.886687 4730 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:44:01.886792 4730 raft_consensus.cc:686] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [term 2 LEADER]: Becoming Leader. State: Replica: 421f41da4efe4d87bfbe161e328ce852, State: Running, Role: LEADER
I20260926 05:44:01.886931 4726 sys_catalog.cc:564] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:01.886950 4730 consensus_queue.cc:237] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } }
I20260926 05:44:01.887311 4732 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 421f41da4efe4d87bfbe161e328ce852. Latest consensus state: current_term: 2 leader_uuid: "421f41da4efe4d87bfbe161e328ce852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } } }
I20260926 05:44:01.887382 4732 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:01.887588 4731 sys_catalog.cc:455] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "421f41da4efe4d87bfbe161e328ce852" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "421f41da4efe4d87bfbe161e328ce852" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40509 } } }
I20260926 05:44:01.887710 4731 sys_catalog.cc:458] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852 [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:01.887827 4735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:01.888398 4735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:01.888549 4735 catalog_manager.cc:1259] Loaded cluster ID: 6ae50806cd014660b0459cb94c161b57
I20260926 05:44:01.888588 4735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:44:01.888728 4726 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 05:44:01.892459 4735 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:01.893123 4735 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Loaded TSK: 0
I20260926 05:44:01.893437 4735 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:44:01.970592 4728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:01.970831 4728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:01.970873 4728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:01.972155 4728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:01.972275 4728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:01.972337 4728 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:44:01.974081 4728 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:44853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=44491
--tserver_master_addrs=127.23.213.254:40509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4728
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:01.974356 4728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:01.974632 4728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:01.980564 4764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.980609 4763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:01.980540 4766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:01.981077 4728 server_base.cc:1034] running on GCE node
I20260926 05:44:01.981281 4728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:01.981530 4728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:01.982681 4728 hybrid_clock.cc:648] HybridClock initialized: now 1790401441982662 us; error 42 us; skew 500 ppm
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:01.985328 4728 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:44:01.985966 4728 webserver.cc:460] Webserver started at http://127.23.213.193:44491/ using document root <none> and password file <none>
I20260926 05:44:01.986194 4728 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:01.986281 4728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:44:01 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:02.006181 4728 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 05:44:02.007225 4774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.007436 4728 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:02.007531 4728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "14de182e1a724ffb83d3fd2d7231b563"
format_stamp: "Formatted at 2026-09-26 05:44:00 on dist-test-slave-1fdt"
server_key: "276aca745d7cfb9cb448984afe11da3d"
server_key_iv: "1f2fb6ccf5c7b6aed8e5898556a89a95"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:02.007948 4728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:02.040503 4728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:02.042834 4728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:02.043021 4728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:02.043339 4728 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:02.043751 4728 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:02.043809 4728 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.043875 4728 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:02.043919 4728 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.051146 4728 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:44853
I20260926 05:44:02.051558 4728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:44:02.052300 4887 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:44853 every 8 connection(s)
I20260926 05:44:02.054749 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 4728
I20260926 05:44:02.055043 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:41507
--local_ip_for_outbound_sockets=127.23.213.194
--tserver_master_addrs=127.23.213.254:40509
--webserver_port=38657
--webserver_interface=127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:44:02 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401441, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:02.072328 4890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:02.052357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40509 (local address 127.23.213.193:55177)
0926 05:44:02.052794 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:02.052844 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:02.053499 (+ 655us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:02.054237 (+ 738us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:02.054241 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:02.054482 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:02.056901 (+ 2419us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:02.056910 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.058898 (+ 1988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.058900 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:02.059997 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:02.060000 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.062569 (+ 2569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.062571 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:02.062706 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:02.065891 (+ 3185us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:02.065903 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:02.067537 (+ 1634us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:02.070259 (+ 2722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:02.070266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:02.070268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:02.070624 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:02.070927 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:02.070928 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:02.070928 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:02.071073 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:02.071623 (+ 550us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:02.071629 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:02.071802 (+ 173us) client_negotiation.cc:769] Sending connection context
0926 05:44:02.071922 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 05:44:02.072085 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":83,"threads_started":1}
I20260926 05:44:02.073082 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:02.052783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:55177 (local address 127.23.213.254:40509)
0926 05:44:02.053004 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:02.053012 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:02.053047 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:44:02.053835 (+ 788us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:02.053843 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:02.053915 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:02.054052 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:02.057046 (+ 2994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.058760 (+ 1714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.061668 (+ 2908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.062345 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.062451 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:02.067732 (+ 5281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:02.067737 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:02.067737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:02.067751 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:02.070150 (+ 2399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.070727 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.070730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.070730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.070845 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.071156 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.071157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.071157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.071491 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:02.071567 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:02.072755 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0926 05:44:02.072876 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":94,"threads_started":1}
I20260926 05:44:02.074278 4888 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40509
I20260926 05:44:02.074410 4888 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.074599 4888 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
W20260926 05:44:02.075546 4888 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:40509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:44:02.075639 4888 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.075709 4888 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:02.076022 4888 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.076094 4888 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
W20260926 05:44:02.076392 4888 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:44:02.170836 4892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:02.171235 4892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:02.171330 4892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:02.173175 4892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:02.173297 4892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:02.173383 4892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:44:02.175712 4892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:41507
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=38657
--tserver_master_addrs=127.23.213.254:40509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4892
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:02.176097 4892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:02.176414 4892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:02.183018 4897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:02.183211 4892 server_base.cc:1034] running on GCE node
W20260926 05:44:02.183010 4900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:02.183012 4898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:02.183612 4892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:02.183914 4892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:02.185109 4892 hybrid_clock.cc:648] HybridClock initialized: now 1790401442185094 us; error 27 us; skew 500 ppm
Sep 26 05:44:02 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:02.188482 4892 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:44:02.189193 4892 webserver.cc:460] Webserver started at http://127.23.213.194:38657/ using document root <none> and password file <none>
I20260926 05:44:02.189503 4892 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:02.189597 4892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:44:02 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:02.214771 4892 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.001s sys 0.000s
I20260926 05:44:02.218228 4908 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.218482 4892 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260926 05:44:02.218604 4892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "b1d43f362b504eb489f8eb3c5685a354"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "b4dbae366e6abf90cc2e8bfff38cd28a"
server_key_iv: "869e2547ba9dcb1e8c367c16ebc4c194"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:02.219273 4892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:02.237476 4892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:02.240150 4892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:02.240327 4892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:02.240619 4892 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:02.241073 4892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:02.241132 4892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.241165 4892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:02.241184 4892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.249420 4892 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:41507
I20260926 05:44:02.249459 5021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:41507 every 8 connection(s)
I20260926 05:44:02.250068 4892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:44:02.254448 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 4892
I20260926 05:44:02.254763 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.23.213.212:49655/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:44407
--local_ip_for_outbound_sockets=127.23.213.195
--tserver_master_addrs=127.23.213.254:40509
--webserver_port=33305
--webserver_interface=127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:44:02 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:02.262394 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:02.250887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:57507 (local address 127.23.213.254:40509)
0926 05:44:02.250955 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:02.250958 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:02.251204 (+ 246us) server_negotiation.cc:407] Connection header received
0926 05:44:02.251943 (+ 739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:02.251945 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:02.251975 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:02.252013 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:02.252956 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.253451 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.255166 (+ 1715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.255513 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.255591 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:02.258377 (+ 2786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:02.258379 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:02.258380 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:02.258389 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:02.260287 (+ 1898us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.260925 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.260927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.260927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.261008 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.261376 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.261377 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.261377 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.261546 (+ 169us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:02.261624 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:02.262083 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 05:44:02.262191 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:44:02.262485 5024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:02.250848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40509 (local address 127.23.213.194:57507)
0926 05:44:02.251038 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:02.251089 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:02.251790 (+ 701us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:02.252232 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:02.252237 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:02.252495 (+ 258us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:02.252820 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:02.252828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.253569 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.253570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:02.255050 (+ 1480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:02.255054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.255649 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.255651 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:02.255818 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:02.256478 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:02.256491 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:02.258247 (+ 1756us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:02.260401 (+ 2154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:02.260408 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:02.260410 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:02.260768 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:02.261100 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:02.261101 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:02.261102 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:02.261273 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:02.261658 (+ 385us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:02.261666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:02.261884 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 05:44:02.262038 (+ 154us) client_negotiation.cc:240] Negotiation successful
0926 05:44:02.262216 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":79,"threads_started":1}
I20260926 05:44:02.263867 5022 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40509
I20260926 05:44:02.263972 5022 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.264183 5022 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
W20260926 05:44:02.265009 5022 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:40509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:44:02.265086 5022 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.265154 5022 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:02.265614 5022 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.265695 5022 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
W20260926 05:44:02.266150 5022 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:44:02.361055 5025 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:02.361440 5025 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:02.361526 5025 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:02.363389 5025 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:02.363511 5025 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:02.363603 5025 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:44:02.365841 5025 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.23.213.212:49655/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:44407
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=33305
--tserver_master_addrs=127.23.213.254:40509
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5025
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:02.366384 5025 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:02.366698 5025 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:02.373178 5030 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:02.373173 5033 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:02.373173 5031 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:02.373536 5025 server_base.cc:1034] running on GCE node
I20260926 05:44:02.373769 5025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:02.374055 5025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:02.381476 5025 hybrid_clock.cc:648] HybridClock initialized: now 1790401442380390 us; error 1091 us; skew 500 ppm
Sep 26 05:44:02 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:02.384845 5025 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:44:02.385396 5025 webserver.cc:460] Webserver started at http://127.23.213.195:33305/ using document root <none> and password file <none>
I20260926 05:44:02.385690 5025 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:02.385792 5025 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 05:44:02 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:02.418161 5025 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.000s
I20260926 05:44:02.419296 5042 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.419519 5025 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:02.419611 5025 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "e8aaeb81322b4939804343ed608cbf5d"
format_stamp: "Formatted at 2026-09-26 05:44:01 on dist-test-slave-1fdt"
server_key: "83d3fbc039bcef18df76b7e475f3e59d"
server_key_iv: "089923c3917e8f335d77d2761c999f97"
server_key_version: "kuduclusterkey@0"
I20260926 05:44:02.420111 5025 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:02.438822 5025 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:02.441407 5025 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:02.441642 5025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:02.441960 5025 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:02.442409 5025 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:02.442468 5025 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.442504 5025 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:02.442574 5025 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:02.450969 5025 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:44407
I20260926 05:44:02.451582 5025 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:44:02.452966 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5025
I20260926 05:44:02.457837 5155 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:44407 every 8 connection(s)
Sep 26 05:44:02 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401442, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:02.471113 4891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:02.453146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:37349 (local address 127.23.213.254:40509)
0926 05:44:02.453208 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:02.453211 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:02.453465 (+ 254us) server_negotiation.cc:407] Connection header received
0926 05:44:02.454420 (+ 955us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:02.454421 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:02.454435 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:02.454474 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:02.455374 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.456139 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.457731 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.458085 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.458129 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:02.460758 (+ 2629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:02.460760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:02.460761 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:02.460767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:02.462485 (+ 1718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.469841 (+ 7356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.469843 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.469843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.469910 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.470256 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.470258 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.470258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.470407 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:02.470468 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:02.470894 (+ 426us) server_negotiation.cc:299] Negotiation successful
0926 05:44:02.470992 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:44:02.473048 5158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:02.452853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40509 (local address 127.23.213.195:37349)
0926 05:44:02.453292 (+ 439us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:02.453339 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:02.454275 (+ 936us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:02.454681 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:02.454686 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:02.454947 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:02.455247 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:02.455254 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.456257 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.456259 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:02.457613 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:02.457616 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.458227 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.458228 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:02.458370 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:02.458992 (+ 622us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:02.459002 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:02.460642 (+ 1640us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:02.469242 (+ 8600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:02.469253 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:02.469256 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:02.469692 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:02.469995 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:02.469997 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:02.469997 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:02.470175 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:02.470519 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:02.470526 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:02.470725 (+ 199us) client_negotiation.cc:769] Sending connection context
0926 05:44:02.472572 (+ 1847us) client_negotiation.cc:240] Negotiation successful
0926 05:44:02.472765 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":121,"threads_started":1}
I20260926 05:44:02.473866 5160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:02.453561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56998 (local address 127.23.213.254:40509)
0926 05:44:02.455407 (+ 1846us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:02.455409 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:02.455417 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:44:02.455440 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:02.455440 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:02.455452 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:02.455523 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:02.458406 (+ 2883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.459086 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.468063 (+ 8977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:02.468381 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:02.468575 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:02.470428 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:02.470429 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:02.470429 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:02.470437 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:02.472206 (+ 1769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.472606 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.472607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.472608 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.472664 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:02.472862 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:02.472863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:02.472864 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:02.472991 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:02.473159 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:02.473664 (+ 505us) server_negotiation.cc:299] Negotiation successful
0926 05:44:02.473773 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1823,"thread_start_us":63,"threads_started":1}
I20260926 05:44:02.478273 5156 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40509
I20260926 05:44:02.478385 5156 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.478595 5156 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
W20260926 05:44:02.479282 5156 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:40509 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 05:44:02.479527 5156 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.479601 5156 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:02.479904 5156 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:02.479984 5156 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
W20260926 05:44:02.480273 5156 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:44:03.076545 4888 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:03.076727 4888 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:03.077049 4690 ts_manager.cc:194] Registered new tserver with Master: 14de182e1a724ffb83d3fd2d7231b563 (127.23.213.193:44853)
I20260926 05:44:03.077952 4690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:55177
Sep 26 05:44:03 dist-test-slave-1fdt krb5kdc[3694](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790401443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:03.266319 5022 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:03.266492 5022 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:03.266785 4690 ts_manager.cc:194] Registered new tserver with Master: b1d43f362b504eb489f8eb3c5685a354 (127.23.213.194:41507)
I20260926 05:44:03.267328 4690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:57507
Sep 26 05:44:03 dist-test-slave-1fdt krb5kdc[3694](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401443, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for HTTP/127.23.213.212@KRBTEST.COM
I20260926 05:44:03.480432 5156 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:03.480585 5156 heartbeater.cc:511] Master 127.23.213.254:40509 requested a full tablet report, sending...
I20260926 05:44:03.480856 4690 ts_manager.cc:194] Registered new tserver with Master: e8aaeb81322b4939804343ed608cbf5d (127.23.213.195:44407)
I20260926 05:44:03.481293 4690 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:37349
I20260926 05:44:03.490142 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:44:03.503875 5178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.491184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57010 (local address 127.23.213.254:40509)
0926 05:44:03.491301 (+ 117us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:03.491304 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:03.491359 (+ 55us) server_negotiation.cc:407] Connection header received
0926 05:44:03.491429 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:03.491430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:03.491442 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:03.491504 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:03.492073 (+ 569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.492568 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.493314 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.493540 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.493643 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.499255 (+ 5612us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:03.499257 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:03.499257 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:03.499265 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:03.502085 (+ 2820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:03.502493 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:03.502494 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:03.502495 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:03.502555 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:03.502946 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:03.502948 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:03.502948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:03.503111 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:03.503222 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:03.503679 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 05:44:03.503766 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":62,"threads_started":1}
I20260926 05:44:03.506348 4690 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:44:03.508729 4690 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:44:03.527823 5183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.520786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:44407 (local address 127.0.0.1:46700)
0926 05:44:03.522301 (+ 1515us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:03.522309 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:03.522347 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:03.522659 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:03.522663 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:03.522666 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:03.522734 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.522737 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.524431 (+ 1694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.524433 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.526381 (+ 1948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.526384 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.527492 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.527493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.527578 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.527592 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:44:03.527679 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:44:03.527732 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1501,"thread_start_us":76,"threads_started":1}
I20260926 05:44:03.528548 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.520935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46700 (local address 127.23.213.195:44407)
0926 05:44:03.521890 (+ 955us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:03.521896 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:03.522402 (+ 506us) server_negotiation.cc:407] Connection header received
0926 05:44:03.522442 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:03.522444 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:03.522461 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:03.522530 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:03.522817 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.524342 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.526484 (+ 2142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.527297 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.527385 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.527445 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:03.528163 (+ 718us) server_negotiation.cc:299] Negotiation successful
0926 05:44:03.528209 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":916,"thread_start_us":63,"threads_started":1}
I20260926 05:44:03.529516 5072 tablet_service.cc:1467] Processing CreateTablet for tablet 2c258aa114ff4dd6a49758ffe1545dfb (DEFAULT_TABLE table=test-table [id=a8880bd3f0514fe797ad8e8c20f2bea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:03.529933 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c258aa114ff4dd6a49758ffe1545dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:03.536372 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.520382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:41507 (local address 127.0.0.1:33222)
0926 05:44:03.521967 (+ 1585us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:03.522010 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:03.522077 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:03.525437 (+ 3360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:03.525438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:03.525440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:03.525496 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.525498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.533580 (+ 8082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.533581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.534769 (+ 1188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.534771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.536120 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.536121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.536202 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.536207 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:44:03.536247 (+ 40us) client_negotiation.cc:240] Negotiation successful
0926 05:44:03.536289 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1556,"thread_start_us":76,"threads_started":1}
I20260926 05:44:03.536921 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.521067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33222 (local address 127.23.213.194:41507)
0926 05:44:03.525187 (+ 4120us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:03.525193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:03.525204 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:44:03.525241 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:03.525243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:03.525260 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:03.525347 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:03.525571 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.532424 (+ 6853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.534878 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.535739 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.535819 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.536749 (+ 930us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:03.536781 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 05:44:03.536829 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4082,"thread_start_us":52,"threads_started":1}
I20260926 05:44:03.537683 4945 tablet_service.cc:1467] Processing CreateTablet for tablet 2c258aa114ff4dd6a49758ffe1545dfb (DEFAULT_TABLE table=test-table [id=a8880bd3f0514fe797ad8e8c20f2bea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:03.538072 4945 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c258aa114ff4dd6a49758ffe1545dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:03.542253 5190 tablet_bootstrap.cc:492] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Bootstrap starting.
I20260926 05:44:03.542937 5190 tablet_bootstrap.cc:654] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:03.543329 5190 log.cc:824] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:03.543731 5189 tablet_bootstrap.cc:492] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Bootstrap starting.
I20260926 05:44:03.544410 5189 tablet_bootstrap.cc:654] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:03.544767 5189 log.cc:824] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:03.545555 5190 tablet_bootstrap.cc:492] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: No bootstrap required, opened a new log
I20260926 05:44:03.545665 5190 ts_tablet_manager.cc:1397] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:44:03.546093 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.520621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44853 (local address 127.0.0.1:34900)
0926 05:44:03.522181 (+ 1560us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:03.522190 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:03.522225 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:03.532758 (+ 10533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:03.532759 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:03.532761 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:03.532805 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.532807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.534156 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.534157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.535328 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.535331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.545836 (+ 10505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.545837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.545926 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.545931 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:44:03.545964 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 05:44:03.546014 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1545,"thread_start_us":68,"threads_started":1}
I20260926 05:44:03.546474 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.521202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34900 (local address 127.23.213.193:44853)
0926 05:44:03.532575 (+ 11373us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:03.532580 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:03.532590 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:44:03.532638 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:03.532640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:03.532656 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:03.532874 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:03.532957 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.534053 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.535438 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.536559 (+ 1121us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.546220 (+ 9661us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.546312 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:03.546343 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 05:44:03.546392 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11337,"thread_start_us":56,"threads_started":1}
I20260926 05:44:03.547111 4819 tablet_service.cc:1467] Processing CreateTablet for tablet 2c258aa114ff4dd6a49758ffe1545dfb (DEFAULT_TABLE table=test-table [id=a8880bd3f0514fe797ad8e8c20f2bea0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:03.547437 4819 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c258aa114ff4dd6a49758ffe1545dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:03.550905 5193 tablet_bootstrap.cc:492] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563: Bootstrap starting.
I20260926 05:44:03.551507 5193 tablet_bootstrap.cc:654] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:03.551844 5193 log.cc:824] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:03.553229 5190 raft_consensus.cc:348] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.553623 5190 raft_consensus.cc:374] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:03.553714 5190 raft_consensus.cc:729] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1d43f362b504eb489f8eb3c5685a354, State: Initialized, Role: FOLLOWER
I20260926 05:44:03.553941 5190 consensus_queue.cc:260] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.554283 5190 ts_tablet_manager.cc:1428] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260926 05:44:03.554436 5022 heartbeater.cc:503] Master 127.23.213.254:40509 was elected leader, sending a full tablet report...
I20260926 05:44:03.556005 5189 tablet_bootstrap.cc:492] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: No bootstrap required, opened a new log
I20260926 05:44:03.556131 5189 ts_tablet_manager.cc:1397] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260926 05:44:03.556171 5193 tablet_bootstrap.cc:492] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563: No bootstrap required, opened a new log
I20260926 05:44:03.556241 5193 ts_tablet_manager.cc:1397] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 05:44:03.558753 5193 raft_consensus.cc:348] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.558825 5189 raft_consensus.cc:348] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.558966 5193 raft_consensus.cc:374] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:03.559008 5193 raft_consensus.cc:729] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14de182e1a724ffb83d3fd2d7231b563, State: Initialized, Role: FOLLOWER
I20260926 05:44:03.559084 5189 raft_consensus.cc:374] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:03.559165 5189 raft_consensus.cc:729] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e8aaeb81322b4939804343ed608cbf5d, State: Initialized, Role: FOLLOWER
I20260926 05:44:03.559155 5193 consensus_queue.cc:260] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.559336 5189 consensus_queue.cc:260] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.559458 5193 ts_tablet_manager.cc:1428] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:44:03.559617 4888 heartbeater.cc:503] Master 127.23.213.254:40509 was elected leader, sending a full tablet report...
I20260926 05:44:03.559631 5189 ts_tablet_manager.cc:1428] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:44:03.559734 5156 heartbeater.cc:503] Master 127.23.213.254:40509 was elected leader, sending a full tablet report...
I20260926 05:44:03.630075 5197 raft_consensus.cc:482] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:03.630249 5197 raft_consensus.cc:504] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.630707 5197 leader_election.cc:288] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1d43f362b504eb489f8eb3c5685a354 (127.23.213.194:41507), 14de182e1a724ffb83d3fd2d7231b563 (127.23.213.193:44853)
I20260926 05:44:03.634948 5187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.631066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:56201 (local address 127.23.213.194:41507)
0926 05:44:03.631114 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:03.631117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:03.631277 (+ 160us) server_negotiation.cc:407] Connection header received
0926 05:44:03.631384 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:03.631385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:03.631395 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:03.631425 (+ 30us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:03.631658 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.632417 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.633971 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.634543 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.634596 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.634605 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:03.634820 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 05:44:03.634861 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:44:03.635443 5198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.630984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:41507 (local address 127.23.213.195:56201)
0926 05:44:03.631193 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:03.631205 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:03.631306 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:03.631498 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:03.631499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:03.631501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:03.631578 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.631581 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.632519 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.632520 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.633851 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.633855 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.634676 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.634677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.634745 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.634748 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:44:03.635278 (+ 530us) client_negotiation.cc:240] Negotiation successful
0926 05:44:03.635350 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":190,"thread_start_us":125,"threads_started":1}
I20260926 05:44:03.635674 5188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.631454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:37885 (local address 127.23.213.193:44853)
0926 05:44:03.631506 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:03.631509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:03.631651 (+ 142us) server_negotiation.cc:407] Connection header received
0926 05:44:03.631788 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:03.631790 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:03.631803 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:03.631840 (+ 37us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:03.632126 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.632893 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.634496 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.635153 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.635219 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.635229 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:03.635543 (+ 314us) server_negotiation.cc:299] Negotiation successful
0926 05:44:03.635592 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:44:03.635927 4957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c258aa114ff4dd6a49758ffe1545dfb" candidate_uuid: "e8aaeb81322b4939804343ed608cbf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1d43f362b504eb489f8eb3c5685a354" is_pre_election: true
I20260926 05:44:03.636046 5199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.631253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44853 (local address 127.23.213.195:37885)
0926 05:44:03.631572 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:03.631584 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:03.631684 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:03.631937 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:03.631938 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:03.631940 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:03.632020 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.632023 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.633022 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.633024 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.634363 (+ 1339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:03.634366 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.635303 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.635304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:03.635434 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.635439 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:44:03.635908 (+ 469us) client_negotiation.cc:240] Negotiation successful
0926 05:44:03.635953 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":16,"thread_start_us":79,"threads_started":1}
I20260926 05:44:03.636093 4957 raft_consensus.cc:2454] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8aaeb81322b4939804343ed608cbf5d in term 0.
I20260926 05:44:03.636313 5045 leader_election.cc:302] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1d43f362b504eb489f8eb3c5685a354, e8aaeb81322b4939804343ed608cbf5d; no voters:
I20260926 05:44:03.636487 5197 raft_consensus.cc:2781] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:44:03.636569 5197 raft_consensus.cc:482] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:44:03.636601 5197 raft_consensus.cc:3037] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:03.636651 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c258aa114ff4dd6a49758ffe1545dfb" candidate_uuid: "e8aaeb81322b4939804343ed608cbf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14de182e1a724ffb83d3fd2d7231b563" is_pre_election: true
I20260926 05:44:03.636790 4842 raft_consensus.cc:2454] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e8aaeb81322b4939804343ed608cbf5d in term 0.
I20260926 05:44:03.637724 5197 raft_consensus.cc:504] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.637933 5197 leader_election.cc:288] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [CANDIDATE]: Term 1 election: Requested vote from peers b1d43f362b504eb489f8eb3c5685a354 (127.23.213.194:41507), 14de182e1a724ffb83d3fd2d7231b563 (127.23.213.193:44853)
I20260926 05:44:03.638180 4957 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c258aa114ff4dd6a49758ffe1545dfb" candidate_uuid: "e8aaeb81322b4939804343ed608cbf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1d43f362b504eb489f8eb3c5685a354"
I20260926 05:44:03.638222 4842 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c258aa114ff4dd6a49758ffe1545dfb" candidate_uuid: "e8aaeb81322b4939804343ed608cbf5d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "14de182e1a724ffb83d3fd2d7231b563"
I20260926 05:44:03.638252 4957 raft_consensus.cc:3037] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:03.638312 4842 raft_consensus.cc:3037] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:03.639070 4957 raft_consensus.cc:2454] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8aaeb81322b4939804343ed608cbf5d in term 1.
I20260926 05:44:03.639287 5045 leader_election.cc:302] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1d43f362b504eb489f8eb3c5685a354, e8aaeb81322b4939804343ed608cbf5d; no voters:
I20260926 05:44:03.639346 4842 raft_consensus.cc:2454] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e8aaeb81322b4939804343ed608cbf5d in term 1.
I20260926 05:44:03.639390 5197 raft_consensus.cc:2781] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:03.639529 5197 raft_consensus.cc:686] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 1 LEADER]: Becoming Leader. State: Replica: e8aaeb81322b4939804343ed608cbf5d, State: Running, Role: LEADER
I20260926 05:44:03.639613 5197 consensus_queue.cc:237] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } }
I20260926 05:44:03.640470 4689 catalog_manager.cc:5495] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d reported cstate change: term changed from 0 to 1, leader changed from <none> to e8aaeb81322b4939804343ed608cbf5d (127.23.213.195). New cstate: current_term: 1 leader_uuid: "e8aaeb81322b4939804343ed608cbf5d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e8aaeb81322b4939804343ed608cbf5d" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 44407 } health_report { overall_health: HEALTHY } } }
I20260926 05:44:03.672585 5186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.669167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46710 (local address 127.23.213.195:44407)
0926 05:44:03.669216 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:03.669219 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:03.669227 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:44:03.669267 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:03.669268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:03.669292 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:03.669362 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:44:03.669583 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.670327 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.671436 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:03.671739 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:03.671794 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:03.671992 (+ 198us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:44:03.672140 (+ 148us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:44:03.672221 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:03.672469 (+ 248us) server_negotiation.cc:299] Negotiation successful
0926 05:44:03.672491 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 05:44:03.675223 4957 raft_consensus.cc:1261] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 1 FOLLOWER]: Refusing update from remote peer e8aaeb81322b4939804343ed608cbf5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:03.675320 4842 raft_consensus.cc:1261] T 2c258aa114ff4dd6a49758ffe1545dfb P 14de182e1a724ffb83d3fd2d7231b563 [term 1 FOLLOWER]: Refusing update from remote peer e8aaeb81322b4939804343ed608cbf5d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:03.675783 5200 consensus_queue.cc:1035] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1d43f362b504eb489f8eb3c5685a354" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 41507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:03.675908 5197 consensus_queue.cc:1035] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "14de182e1a724ffb83d3fd2d7231b563" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:03.678188 5203 mvcc.cc:205] Tried to move back new op lower bound from 7333484313289887744 to 7333484313148342272. Current Snapshot: MvccSnapshot[applied={T|T < 7333484313289887744}]
I20260926 05:44:03.680214 5205 mvcc.cc:205] Tried to move back new op lower bound from 7333484313289887744 to 7333484313148342272. Current Snapshot: MvccSnapshot[applied={T|T < 7333484313289887744}]
I20260926 05:44:03.680308 5204 mvcc.cc:205] Tried to move back new op lower bound from 7333484313289887744 to 7333484313148342272. Current Snapshot: MvccSnapshot[applied={T|T < 7333484313289887744}]
I20260926 05:44:03.682783 4689 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:03.682885 4689 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:57010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:03.683638 4689 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 421f41da4efe4d87bfbe161e328ce852: Sending DeleteTablet for 3 replicas of tablet 2c258aa114ff4dd6a49758ffe1545dfb
I20260926 05:44:03.683995 5072 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c258aa114ff4dd6a49758ffe1545dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:03 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:46700
I20260926 05:44:03.684072 4945 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c258aa114ff4dd6a49758ffe1545dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:03 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:33222
I20260926 05:44:03.684087 4819 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c258aa114ff4dd6a49758ffe1545dfb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:03 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:34900
I20260926 05:44:03.684662 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 4728
I20260926 05:44:03.689865 5214 tablet_replica.cc:330] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: stopping tablet replica
I20260926 05:44:03.690032 5214 raft_consensus.cc:2229] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:03.690191 5214 raft_consensus.cc:2258] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:03.691202 5214 ts_tablet_manager.cc:1905] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:03.693611 5214 ts_tablet_manager.cc:1918] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:03.693760 5214 log.cc:1196] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/2c258aa114ff4dd6a49758ffe1545dfb
I20260926 05:44:03.693869 5213 tablet_replica.cc:330] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: stopping tablet replica
I20260926 05:44:03.694054 5213 raft_consensus.cc:2229] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:44:03.694121 5214 ts_tablet_manager.cc:1939] T 2c258aa114ff4dd6a49758ffe1545dfb P b1d43f362b504eb489f8eb3c5685a354: Deleting consensus metadata
I20260926 05:44:03.694270 5213 raft_consensus.cc:2258] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:03.694615 4676 catalog_manager.cc:4841] TS b1d43f362b504eb489f8eb3c5685a354 (127.23.213.194:41507): tablet 2c258aa114ff4dd6a49758ffe1545dfb (table test-table [id=a8880bd3f0514fe797ad8e8c20f2bea0]) successfully deleted
I20260926 05:44:03.694804 5213 ts_tablet_manager.cc:1905] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 05:44:03.695127 4676 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:44853 (error 108)
I20260926 05:44:03.696092 5185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:03.695626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44853 (local address 127.0.0.1:34916)
0926 05:44:03.696014 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:03.696058 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:44853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366}
W20260926 05:44:03.696213 4676 catalog_manager.cc:4584] TS 14de182e1a724ffb83d3fd2d7231b563 (127.23.213.193:44853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c258aa114ff4dd6a49758ffe1545dfb: Network error: Client connection negotiation failed: client connection to 127.23.213.193:44853: connect: Connection refused (error 111)
I20260926 05:44:03.705925 5213 ts_tablet_manager.cc:1918] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:03.706029 5213 log.cc:1196] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/2c258aa114ff4dd6a49758ffe1545dfb
I20260926 05:44:03.706362 5213 ts_tablet_manager.cc:1939] T 2c258aa114ff4dd6a49758ffe1545dfb P e8aaeb81322b4939804343ed608cbf5d: Deleting consensus metadata
I20260926 05:44:03.707000 4676 catalog_manager.cc:4841] TS e8aaeb81322b4939804343ed608cbf5d (127.23.213.195:44407): tablet 2c258aa114ff4dd6a49758ffe1545dfb (table test-table [id=a8880bd3f0514fe797ad8e8c20f2bea0]) successfully deleted
I20260926 05:44:03.709978 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 4892
I20260926 05:44:03.719275 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5025
I20260926 05:44:03.724766 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 4657
I20260926 05:44:03.730545 24407 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 05:44:04.075103 24407 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 05:44:04.075199 24407 mini_ranger.cc:66] Stopping Ranger...
I20260926 05:44:04.420377 24407 mini_ranger.cc:68] Stopped Ranger
2026-09-26 05:44:04.420 UTC [3724] LOG: received smart shutdown request
2026-09-26 05:44:04.424 UTC [3724] LOG: background worker "logical replication launcher" (PID 3733) exited with exit code 1
2026-09-26 05:44:04.425 UTC [3728] LOG: shutting down
2026-09-26 05:44:04.612 UTC [3724] LOG: database system is shut down
2026-09-26T05:44:04Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (48085 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:44:04 dist-test-slave-1fdt krb5kdc[5225](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:44:04 dist-test-slave-1fdt krb5kdc[5225](info): set up 2 sockets
Sep 26 05:44:04 dist-test-slave-1fdt krb5kdc[5225](info): commencing operation
krb5kdc: starting...
W20260926 05:44:05.770388 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:44:05 dist-test-slave-1fdt krb5kdc[5225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:44:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:44:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:44:05.892875 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:39273
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:39273,127.23.213.253:33439,127.23.213.252:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:05.973016 5238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:05.973227 5238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:05.973249 5238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:05.974329 5238 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:05.974363 5238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:05.974377 5238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:05.974390 5238 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:05.974401 5238 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:05.975821 5238 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:39273,127.23.213.253:33439,127.23.213.252:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:39273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:05.976008 5238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:05.976172 5238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:05.981073 5244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:05.981102 5246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:05.981102 5243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:05.981448 5238 server_base.cc:1034] running on GCE node
I20260926 05:44:05.981802 5238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:05.982071 5238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:05.983217 5238 hybrid_clock.cc:648] HybridClock initialized: now 1790401445983203 us; error 29 us; skew 500 ppm
Sep 26 05:44:05 dist-test-slave-1fdt krb5kdc[5225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:05.985651 5238 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:05.986110 5238 webserver.cc:460] Webserver started at http://127.23.213.254:36349/ using document root <none> and password file <none>
I20260926 05:44:05.986337 5238 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:05.986382 5238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:05.986526 5238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:05.987618 5238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "b20a6589c89c41ce887898bdc9bca165"
format_stamp: "Formatted at 2026-09-26 05:44:05 on dist-test-slave-1fdt"
server_key: "5e61f3dfbe312c3e3b3005f85fa48128"
server_key_iv: "8eeee4d3571ca730fd7bd05409b0eaba"
server_key_version: "encryptionkey@0"
I20260926 05:44:05.987993 5238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "b20a6589c89c41ce887898bdc9bca165"
format_stamp: "Formatted at 2026-09-26 05:44:05 on dist-test-slave-1fdt"
server_key: "5e61f3dfbe312c3e3b3005f85fa48128"
server_key_iv: "8eeee4d3571ca730fd7bd05409b0eaba"
server_key_version: "encryptionkey@0"
I20260926 05:44:05.989356 5238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:44:05.990144 5253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:05.990314 5238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:05.990423 5238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "b20a6589c89c41ce887898bdc9bca165"
format_stamp: "Formatted at 2026-09-26 05:44:05 on dist-test-slave-1fdt"
server_key: "5e61f3dfbe312c3e3b3005f85fa48128"
server_key_iv: "8eeee4d3571ca730fd7bd05409b0eaba"
server_key_version: "encryptionkey@0"
I20260926 05:44:05.990502 5238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:06.005620 5238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:06.007575 5238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:06.007758 5238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:06.011910 5238 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:39273
I20260926 05:44:06.011945 5305 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:39273 every 8 connection(s)
I20260926 05:44:06.012307 5238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:06.013207 5306 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:06.014801 5306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:44:06.017650 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5238
I20260926 05:44:06.017777 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:44:06.017946 24407 external_mini_cluster.cc:1428] Setting key 744bd9f5941b0614111a2fd2758eab02
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:06.025031 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.015428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39273 (local address 127.0.0.1:51112)
0926 05:44:06.015603 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.015637 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.016114 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.016415 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.016418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.016694 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.016845 (+ 151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.016849 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.018144 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.018145 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.018812 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.018814 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.019446 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.019447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.019534 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.020048 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.020059 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.021658 (+ 1599us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.023505 (+ 1847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.023511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.023514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.023710 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.023970 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.023971 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.023971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.024057 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.024422 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.024427 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.024489 (+ 62us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.024613 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.024852 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":97,"thread_start_us":85,"threads_started":1}
I20260926 05:44:06.025033 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.015600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.23.213.254:39273)
0926 05:44:06.015723 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.015728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.015765 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:44:06.016278 (+ 513us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.016280 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.016311 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.016410 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.016952 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.018037 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.018902 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.019350 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.019431 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.021792 (+ 2361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.021799 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.021799 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.021810 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.023401 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.023811 (+ 410us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.023813 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.023814 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.023894 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.024134 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.024134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.024135 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.024347 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.024443 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.024648 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.024788 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":47,"threads_started":1}
WARNING: no policy specified for kudu/127.23.213.253@KRBTEST.COM; defaulting to no policy
I20260926 05:44:06.025805 5306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:44:06.026156 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.026043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43908)
0926 05:44:06.026093 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.026136 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:44:06.026292 5257 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
I20260926 05:44:06.026705 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.026605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43916)
0926 05:44:06.026658 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.026683 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 05:44:06.026811 5306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:33439: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab.
Entry for principal kudu/127.23.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab.
I20260926 05:44:06.045179 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:33439
--webserver_interface=127.23.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab
--principal=kudu/127.23.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:39273,127.23.213.253:33439,127.23.213.252:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:06.076063 5306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } attempt: 1
I20260926 05:44:06.076607 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.076458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43920)
0926 05:44:06.076537 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.076577 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20260926 05:44:06.076959 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.076859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43936)
0926 05:44:06.076910 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.076934 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:44:06.077045 5306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:33439: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
W20260926 05:44:06.127663 5313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:06.127875 5313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:06.127897 5313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:06.129060 5313 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:06.129102 5313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:06.129117 5313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:06.129129 5313 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:06.129141 5313 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:06.130775 5313 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:39273,127.23.213.253:33439,127.23.213.252:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab
--principal=kudu/127.23.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:33439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5313
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:06.131035 5313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:06.131265 5313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:06.136315 5319 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:06.136458 5313 server_base.cc:1034] running on GCE node
W20260926 05:44:06.136372 5321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:06.136586 5318 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:06.136874 5313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:06.137198 5313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:06.138410 5313 hybrid_clock.cc:648] HybridClock initialized: now 1790401446138397 us; error 33 us; skew 500 ppm
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:06.141371 5313 init.cc:377] Logged in from keytab as kudu/127.23.213.253@KRBTEST.COM (short username kudu)
I20260926 05:44:06.141973 5313 webserver.cc:460] Webserver started at http://127.23.213.253:44791/ using document root <none> and password file <none>
I20260926 05:44:06.142225 5313 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:06.142275 5313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:06.142467 5313 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:06.143663 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data/instance:
uuid: "e6ba4f1654814b4595f28acd5bb7f1fe"
format_stamp: "Formatted at 2026-09-26 05:44:06 on dist-test-slave-1fdt"
server_key: "d5156f63e06fa0b066c29a8de09b291c"
server_key_iv: "2489d2dec1dfe7ea4b559dc1ad07bbe6"
server_key_version: "encryptionkey@0"
I20260926 05:44:06.144069 5313 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/wal/instance:
uuid: "e6ba4f1654814b4595f28acd5bb7f1fe"
format_stamp: "Formatted at 2026-09-26 05:44:06 on dist-test-slave-1fdt"
server_key: "d5156f63e06fa0b066c29a8de09b291c"
server_key_iv: "2489d2dec1dfe7ea4b559dc1ad07bbe6"
server_key_version: "encryptionkey@0"
I20260926 05:44:06.145174 5306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } attempt: 2
I20260926 05:44:06.145542 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.145422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43948)
0926 05:44:06.145482 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.145511 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:44:06.145562 5313 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:44:06.145972 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.145876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43954)
0926 05:44:06.145932 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.145957 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20260926 05:44:06.146100 5306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:33439: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33439: connect: Connection refused (error 111)
I20260926 05:44:06.146639 5328 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:06.146853 5313 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:44:06.146971 5313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/wal
uuid: "e6ba4f1654814b4595f28acd5bb7f1fe"
format_stamp: "Formatted at 2026-09-26 05:44:06 on dist-test-slave-1fdt"
server_key: "d5156f63e06fa0b066c29a8de09b291c"
server_key_iv: "2489d2dec1dfe7ea4b559dc1ad07bbe6"
server_key_version: "encryptionkey@0"
I20260926 05:44:06.147054 5313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:06.163527 5313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:06.166636 5313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:06.166805 5313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:06.171283 5313 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.253:33439
I20260926 05:44:06.171522 5380 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.253:33439 every 8 connection(s)
I20260926 05:44:06.171720 5313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
I20260926 05:44:06.172753 5381 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:06.174633 5381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:44:06.180403 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5313
I20260926 05:44:06.180505 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-1/wal/instance
I20260926 05:44:06.180737 24407 external_mini_cluster.cc:1428] Setting key ff3f4549ca458a9a4ce8b0a7cab10336
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:06.185380 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.175523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51128 (local address 127.23.213.254:39273)
0926 05:44:06.175592 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.175594 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.175653 (+ 59us) server_negotiation.cc:407] Connection header received
0926 05:44:06.176105 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.176107 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.176121 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.176192 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.176825 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.177433 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.178474 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.178697 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.178790 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.181432 (+ 2642us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.181433 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.181434 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.181442 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.183709 (+ 2267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.184253 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.184254 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.184255 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.184327 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.184631 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.184632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.184632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.184777 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.184889 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.185178 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.185276 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:44:06.185642 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.175351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39273 (local address 127.0.0.1:51128)
0926 05:44:06.175518 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.175561 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.176005 (+ 444us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.176266 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.176269 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.176513 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.176722 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.176726 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.177528 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.177528 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.178375 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.178377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.178800 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.178801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.178926 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.179418 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.179426 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.181316 (+ 1890us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.183806 (+ 2490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.183812 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.183814 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.184126 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.184416 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.184417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.184417 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.184538 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.184857 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.184864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.185030 (+ 166us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.185157 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.185338 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
I20260926 05:44:06.186247 5381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.213.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.252@KRBTEST.COM" created.
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:44:06.195663 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.186557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43958)
0926 05:44:06.186789 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.186801 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.186893 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.187056 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.187057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.187207 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.187255 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.187257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.188223 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.188224 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.188898 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.188900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.189416 (+ 516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.189417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.189489 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.189908 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.189913 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.192181 (+ 2268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.194482 (+ 2301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.194483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.194484 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.194659 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.194929 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.194930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.194930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.194990 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.195310 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.195311 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.195352 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.195433 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.195541 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 05:44:06.195766 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.186645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43958 (local address 127.23.213.253:33439)
0926 05:44:06.186809 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.186814 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.186890 (+ 76us) server_negotiation.cc:407] Connection header received
0926 05:44:06.186952 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.186954 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.186968 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.187036 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.187340 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.188101 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.188984 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.189311 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.189416 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.192294 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.192302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.192302 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.192309 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.194381 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.194766 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.194770 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.194770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.194835 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.195067 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.195069 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.195069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.195228 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.195334 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.195484 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.195616 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":133,"thread_start_us":104,"threads_started":1}
I20260926 05:44:06.196343 5381 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:44:06.196704 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.196599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46130)
0926 05:44:06.196640 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.196683 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260926 05:44:06.196878 5332 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
I20260926 05:44:06.197295 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.197231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46134)
0926 05:44:06.197256 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.197280 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260926 05:44:06.197424 5381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35513: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab.
Entry for principal kudu/127.23.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab.
I20260926 05:44:06.209087 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:35513
--webserver_interface=127.23.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab
--principal=kudu/127.23.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:39273,127.23.213.253:33439,127.23.213.252:35513
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:06.237252 5306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } attempt: 3
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:44:06.244421 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.237688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.23.213.253:33439)
0926 05:44:06.237779 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.237782 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.237793 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:44:06.237851 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.237852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.237866 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.237922 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.238266 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.238809 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.239557 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.239787 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.239879 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.241785 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.241787 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.241787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.241796 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.243257 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.243629 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.243630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.243631 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.243689 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.243891 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.243892 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.243893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.244008 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.244086 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.244240 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.244321 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:44:06.244421 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.237606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43962)
0926 05:44:06.237680 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.237697 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.237782 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.237929 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.237930 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.238094 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.238158 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.238160 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.238899 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.238900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.239442 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.239444 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.239880 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.239881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.239974 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.240465 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.240470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.241676 (+ 1206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.243364 (+ 1688us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.243365 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.243365 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.243537 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.243760 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.243761 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.243761 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.243824 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.244080 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.244081 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.244128 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.244217 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.244320 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20260926 05:44:06.244791 5306 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:44:06.245163 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.245060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46140)
0926 05:44:06.245113 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.245144 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:44:06.245525 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.245415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46150)
0926 05:44:06.245485 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.245509 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260926 05:44:06.245644 5306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35513: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
I20260926 05:44:06.246630 5381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } attempt: 1
I20260926 05:44:06.247008 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.246901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46160)
0926 05:44:06.246959 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.246987 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 05:44:06.247293 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.247239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46166)
0926 05:44:06.247260 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.247281 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260926 05:44:06.247393 5381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35513: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
I20260926 05:44:06.276801 5306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } attempt: 1
I20260926 05:44:06.277289 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.277148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46182)
0926 05:44:06.277222 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.277261 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 05:44:06.277683 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.277582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46186)
0926 05:44:06.277633 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.277657 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:44:06.277812 5306 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35513: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
W20260926 05:44:06.292142 5388 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:06.292379 5388 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:06.292418 5388 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:06.293648 5388 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:06.293710 5388 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:06.293733 5388 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:06.293751 5388 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:06.293823 5388 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:06.295276 5388 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:39273,127.23.213.253:33439,127.23.213.252:35513
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab
--principal=kudu/127.23.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:35513
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5388
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:06.295550 5388 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:06.295786 5388 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:06.300729 5394 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:06.300688 5393 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:06.300697 5396 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:06.301023 5388 server_base.cc:1034] running on GCE node
I20260926 05:44:06.301290 5388 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:06.301587 5388 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:06.302759 5388 hybrid_clock.cc:648] HybridClock initialized: now 1790401446302754 us; error 40 us; skew 500 ppm
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:06.305285 5388 init.cc:377] Logged in from keytab as kudu/127.23.213.252@KRBTEST.COM (short username kudu)
I20260926 05:44:06.305720 5388 webserver.cc:460] Webserver started at http://127.23.213.252:35749/ using document root <none> and password file <none>
I20260926 05:44:06.305982 5388 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:06.306057 5388 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:06.306213 5388 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:06.307281 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data/instance:
uuid: "c1442b54fbc3492996df6736910000fa"
format_stamp: "Formatted at 2026-09-26 05:44:06 on dist-test-slave-1fdt"
server_key: "a07afdab880656bb851bee941562bba2"
server_key_iv: "41fef37c6fc6f88df2d15d6c314de0bb"
server_key_version: "encryptionkey@0"
I20260926 05:44:06.307646 5388 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/wal/instance:
uuid: "c1442b54fbc3492996df6736910000fa"
format_stamp: "Formatted at 2026-09-26 05:44:06 on dist-test-slave-1fdt"
server_key: "a07afdab880656bb851bee941562bba2"
server_key_iv: "41fef37c6fc6f88df2d15d6c314de0bb"
server_key_version: "encryptionkey@0"
I20260926 05:44:06.309008 5388 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:44:06.309849 5403 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:06.310027 5388 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:44:06.310146 5388 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/wal
uuid: "c1442b54fbc3492996df6736910000fa"
format_stamp: "Formatted at 2026-09-26 05:44:06 on dist-test-slave-1fdt"
server_key: "a07afdab880656bb851bee941562bba2"
server_key_iv: "41fef37c6fc6f88df2d15d6c314de0bb"
server_key_version: "encryptionkey@0"
I20260926 05:44:06.310242 5388 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:06.315524 5381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } attempt: 2
I20260926 05:44:06.315932 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.315803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46192)
0926 05:44:06.315863 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.315898 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:44:06.316258 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.316168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46200)
0926 05:44:06.316213 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.316236 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:44:06.316360 5381 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35513: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35513: connect: Connection refused (error 111)
I20260926 05:44:06.319614 5388 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:06.322072 5388 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:06.322237 5388 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:06.326327 5388 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.252:35513
I20260926 05:44:06.326352 5455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.252:35513 every 8 connection(s)
I20260926 05:44:06.326730 5388 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
I20260926 05:44:06.327576 5456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:06.329123 5456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:44:06.333976 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5388
I20260926 05:44:06.334096 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790401292175461-24407-0/minicluster-data/master-2/wal/instance
I20260926 05:44:06.334298 24407 external_mini_cluster.cc:1428] Setting key 8a50d781a22c7c91af31c4be3f489188
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:06.338619 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.329867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51136 (local address 127.23.213.254:39273)
0926 05:44:06.329934 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.329936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.330158 (+ 222us) server_negotiation.cc:407] Connection header received
0926 05:44:06.330693 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.330695 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.330708 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.330769 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.331380 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.331862 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.333008 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.333221 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.333312 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.335861 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.335862 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.335863 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.335869 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.337205 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.337610 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.337611 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.337612 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.337674 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.337960 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.337961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.337962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.338081 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.338161 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.338441 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.338532 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:44:06.338805 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.329859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:39273 (local address 127.0.0.1:51136)
0926 05:44:06.330033 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.330067 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.330551 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.330845 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.330849 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.331087 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.331275 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.331279 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.331968 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.331969 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.332900 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.332902 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.333319 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.333320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.333449 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.334036 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.334049 (+ 13us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.335779 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.337277 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.337282 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.337284 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.337493 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.337774 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.337775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.337776 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.337880 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.338165 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.338172 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.338300 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.338407 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.338552 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":77,"threads_started":1}
I20260926 05:44:06.339344 5456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:44:06.343670 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.334701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46204 (local address 127.23.213.252:35513)
0926 05:44:06.334862 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.334866 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.334974 (+ 108us) server_negotiation.cc:407] Connection header received
0926 05:44:06.334996 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.334998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.335018 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.335103 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.335786 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.336727 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.338029 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.338346 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.338429 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.340924 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.340932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.340933 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.340939 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.342436 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.342797 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.342800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.342800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.342858 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.343079 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.343080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.343080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.343231 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.343407 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.343455 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.343546 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":99,"threads_started":1}
I20260926 05:44:06.346547 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.339683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43978 (local address 127.23.213.253:33439)
0926 05:44:06.339740 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.339742 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.339774 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:44:06.339837 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.339839 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.339849 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.339907 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.340175 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.340711 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.341560 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.341886 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.341990 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.343753 (+ 1763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.343755 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.343755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.343763 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.345309 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.345733 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.345735 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.345735 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.345820 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.346032 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.346034 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.346034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.346150 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.346224 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.346382 (+ 158us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.346455 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 05:44:06.346549 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.339619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33439 (local address 127.0.0.1:43978)
0926 05:44:06.339727 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.339741 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.339783 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.339908 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.339908 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.340031 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.340078 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.340079 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.340791 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.340793 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.341453 (+ 660us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.341456 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.341990 (+ 534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.341991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.342062 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.342458 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.342463 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.343661 (+ 1198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.345400 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.345401 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.345401 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.345619 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.345897 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.345898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.345898 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.345957 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.346231 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.346231 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.346272 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.346357 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.346455 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92}
I20260926 05:44:06.347254 5456 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:44:06.352273 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.344830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51150 (local address 127.23.213.254:39273)
0926 05:44:06.344890 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.344892 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.344898 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:44:06.344914 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.344915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.344923 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.344984 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.345451 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.346002 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.346707 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.347018 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.347110 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.349277 (+ 2167us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.349279 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.349279 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.349287 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.350916 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.351359 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.351360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.351360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.351418 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.351649 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.351649 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.351649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.351750 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.351924 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.351963 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.352068 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 05:44:06.352991 5306 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } attempt: 2
I20260926 05:44:06.354451 5458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.347513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46208)
0926 05:44:06.347551 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.347564 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.347608 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.347730 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.347731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.347860 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.347883 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.347885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.348508 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.348510 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.349030 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.349032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.349412 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.349412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.349461 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.349966 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.349971 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.351427 (+ 1456us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.353365 (+ 1938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.353366 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.353366 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.353566 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.353794 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.353794 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.353795 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.353857 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.354132 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.354133 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.354176 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.354244 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.354333 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 05:44:06.354568 5460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.347546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46208 (local address 127.23.213.252:35513)
0926 05:44:06.347584 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.347586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.347594 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:44:06.347664 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.347665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.347675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.347728 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.347982 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.348426 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.349121 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.349326 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.349411 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.351514 (+ 2103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.351515 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.351515 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.351522 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.353285 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.353633 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.353634 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.353634 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.353683 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.353941 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.353942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.353942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.354075 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.354143 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.354288 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.354405 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":22}
I20260926 05:44:06.356171 5456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa: Bootstrap starting.
I20260926 05:44:06.356976 5456 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:06.357345 5456 log.cc:824] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa: Log is configured to *not* fsync() on all Append() calls
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:44:06.358227 5456 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa: No bootstrap required, opened a new log
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401445, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:44:06.360148 5456 raft_consensus.cc:348] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.360258 5456 raft_consensus.cc:374] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:06.360280 5456 raft_consensus.cc:729] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1442b54fbc3492996df6736910000fa, State: Initialized, Role: FOLLOWER
I20260926 05:44:06.360425 5456 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.360719 5456 sys_catalog.cc:564] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:06.360800 5463 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.360882 5463 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:44:06.362556 5474 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:44:06.362609 5474 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:44:06.362630 5308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.353234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46224)
0926 05:44:06.353316 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.353326 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.353360 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.353816 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.353817 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.353972 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.354012 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.354013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.355740 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.355741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.356511 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.356513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.356932 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.356933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.356989 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.357397 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.357404 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.359091 (+ 1687us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.361390 (+ 2299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.361391 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.361392 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.361627 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.361892 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.361893 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.361893 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.361957 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.362297 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.362299 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.362343 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.362411 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.362504 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260926 05:44:06.363108 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.353443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.23.213.252:35513)
0926 05:44:06.353629 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.353631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.353639 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:44:06.353661 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.353661 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.353672 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.353806 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.354089 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.355640 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.356600 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.356834 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.356930 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.359177 (+ 2247us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.359178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.359178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.359184 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.361303 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.361722 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.361724 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.361724 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.361809 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.362035 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.362036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.362037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.362202 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.362269 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.362813 (+ 544us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.362887 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":108,"threads_started":1}
I20260926 05:44:06.363400 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.353041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43980 (local address 127.23.213.253:33439)
0926 05:44:06.353078 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.353080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.353223 (+ 143us) server_negotiation.cc:407] Connection header received
0926 05:44:06.353290 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.353291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.353300 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.353343 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.353995 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.354664 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.355714 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.355981 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.356050 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.359623 (+ 3573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.359625 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.359625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.359633 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.362167 (+ 2534us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.362526 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.362527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.362528 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.362581 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.362793 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.362794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.362795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.362958 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.363155 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.363241 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.363315 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:44:06.364521 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:44:06.364678 5306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: Bootstrap starting.
I20260926 05:44:06.365373 5306 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:06.363255 5388 master_runner.cc:186] Error getting master registration for 127.23.213.254:39273: OK, instance_id { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" instance_seqno: 1790401446010727 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:44:06.365561 5388 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:44:06.365680 5306 log.cc:824] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:06.366541 5306 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: No bootstrap required, opened a new log
I20260926 05:44:06.368508 5306 raft_consensus.cc:348] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.368643 5306 raft_consensus.cc:374] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:06.368690 5306 raft_consensus.cc:729] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b20a6589c89c41ce887898bdc9bca165, State: Initialized, Role: FOLLOWER
I20260926 05:44:06.368809 5306 consensus_queue.cc:260] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.369115 5306 sys_catalog.cc:564] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:06.370024 5484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.370123 5484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:44:06.375890 5385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.366073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43986 (local address 127.23.213.253:33439)
0926 05:44:06.366116 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.366118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.366538 (+ 420us) server_negotiation.cc:407] Connection header received
0926 05:44:06.366626 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.366627 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.366636 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.366666 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.367324 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.367989 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.368932 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.369221 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.369258 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.371858 (+ 2600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.371859 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.371860 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.371866 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.374132 (+ 2266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.374516 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.374517 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.374517 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.374580 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.375121 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.375122 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.375123 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.375264 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.375339 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.375687 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.375765 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:44:06.372979 5238 master_runner.cc:186] Error getting master registration for 127.23.213.253:33439: OK, instance_id { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" instance_seqno: 1790401446170071 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:44:06.375967 5238 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 05:44:06.376953 5495 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:44:06.377008 5495 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:44:06.377190 5309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.365784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51152 (local address 127.23.213.254:39273)
0926 05:44:06.365819 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.365821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.366252 (+ 431us) server_negotiation.cc:407] Connection header received
0926 05:44:06.366355 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.366356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.366365 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.366397 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.367176 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.367924 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.368932 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.369219 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.369258 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.371022 (+ 1764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.371024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.371025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.371031 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.373376 (+ 2345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.375530 (+ 2154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.375531 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.375532 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.375584 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.376561 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.376562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.376562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.376703 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.376797 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.376998 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.377074 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:44:06.378468 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.366303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46240 (local address 127.23.213.252:35513)
0926 05:44:06.366337 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.366339 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.370296 (+ 3957us) server_negotiation.cc:407] Connection header received
0926 05:44:06.370399 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.370400 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.370411 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.370444 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.371390 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.372133 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.373570 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.373956 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.374003 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.375448 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.375449 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.375450 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.375457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.377158 (+ 1701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.377634 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.377636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.377636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.377690 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.377939 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.377940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.377940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.378055 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.378198 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.378298 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.378372 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:44:06.407516 5381 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } attempt: 3
Sep 26 05:44:06 dist-test-slave-1fdt krb5kdc[5225](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401446, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:44:06.416138 5461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.407951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46256 (local address 127.23.213.252:35513)
0926 05:44:06.407994 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:06.407996 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:06.408003 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:44:06.408025 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:06.408025 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:06.408034 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:06.408087 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:06.408512 (+ 425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.409272 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.410311 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.410607 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.410682 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.413010 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:06.413012 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:06.413013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:06.413020 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:06.414835 (+ 1815us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.415264 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.415265 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.415265 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.415327 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:06.415565 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:06.415565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:06.415566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:06.415708 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:06.415777 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:06.415961 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 05:44:06.416041 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:44:06.416143 5383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:06.407821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35513 (local address 127.0.0.1:46256)
0926 05:44:06.407869 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:06.407880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:06.407918 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:06.408129 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:06.408130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:06.408342 (+ 212us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:06.408403 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.408406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.409361 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.409362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.410202 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:06.410205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:06.410694 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:06.410695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:06.410763 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:06.411301 (+ 538us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:06.411306 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:06.412914 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:06.414933 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.414934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.414934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.415177 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.415407 (+ 230us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:06.415408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:06.415408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:06.415488 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:06.415785 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:06.415786 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:06.415837 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 05:44:06.415924 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 05:44:06.416028 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 05:44:06.418484 5381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: Bootstrap starting.
I20260926 05:44:06.419229 5381 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:06.419523 5381 log.cc:824] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:06.420449 5381 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: No bootstrap required, opened a new log
I20260926 05:44:06.422601 5381 raft_consensus.cc:348] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.422755 5381 raft_consensus.cc:374] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:06.422799 5381 raft_consensus.cc:729] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e6ba4f1654814b4595f28acd5bb7f1fe, State: Initialized, Role: FOLLOWER
I20260926 05:44:06.422930 5381 consensus_queue.cc:260] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.423233 5498 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.423295 5381 sys_catalog.cc:564] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:06.423389 5498 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:44:06.425302 5509 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:44:06.425405 5509 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:44:06.427778 5313 master_runner.cc:263] No leader master found from master e6ba4f1654814b4595f28acd5bb7f1fe
I20260926 05:44:06.427839 5313 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 05:44:06.508632 5259 tablet.cc:2369] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:44:06.572868 5409 tablet.cc:2369] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:44:06.667788 5334 tablet.cc:2369] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:44:06.682411 5463 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:06.682542 5463 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.682852 5463 leader_election.cc:288] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b20a6589c89c41ce887898bdc9bca165 (127.23.213.254:39273), e6ba4f1654814b4595f28acd5bb7f1fe (127.23.213.253:33439)
I20260926 05:44:06.683352 5280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1442b54fbc3492996df6736910000fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20a6589c89c41ce887898bdc9bca165" is_pre_election: true
I20260926 05:44:06.683439 5355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1442b54fbc3492996df6736910000fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" is_pre_election: true
I20260926 05:44:06.683562 5280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1442b54fbc3492996df6736910000fa in term 0.
I20260926 05:44:06.683598 5355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c1442b54fbc3492996df6736910000fa in term 0.
I20260926 05:44:06.683789 5407 leader_election.cc:302] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20a6589c89c41ce887898bdc9bca165, c1442b54fbc3492996df6736910000fa; no voters:
I20260926 05:44:06.683961 5463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:44:06.684051 5463 raft_consensus.cc:482] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:44:06.684094 5463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:06.684803 5463 raft_consensus.cc:504] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.684947 5463 leader_election.cc:288] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [CANDIDATE]: Term 1 election: Requested vote from peers b20a6589c89c41ce887898bdc9bca165 (127.23.213.254:39273), e6ba4f1654814b4595f28acd5bb7f1fe (127.23.213.253:33439)
I20260926 05:44:06.685153 5280 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1442b54fbc3492996df6736910000fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b20a6589c89c41ce887898bdc9bca165"
I20260926 05:44:06.685213 5355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c1442b54fbc3492996df6736910000fa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe"
I20260926 05:44:06.685261 5280 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:06.685343 5355 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:06.686131 5280 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1442b54fbc3492996df6736910000fa in term 1.
I20260926 05:44:06.686269 5355 raft_consensus.cc:2454] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c1442b54fbc3492996df6736910000fa in term 1.
I20260926 05:44:06.686373 5407 leader_election.cc:302] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b20a6589c89c41ce887898bdc9bca165, c1442b54fbc3492996df6736910000fa; no voters:
I20260926 05:44:06.686456 5463 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:06.686626 5463 raft_consensus.cc:686] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [term 1 LEADER]: Becoming Leader. State: Replica: c1442b54fbc3492996df6736910000fa, State: Running, Role: LEADER
I20260926 05:44:06.686731 5463 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } }
I20260926 05:44:06.687258 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1442b54fbc3492996df6736910000fa. Latest consensus state: current_term: 1 leader_uuid: "c1442b54fbc3492996df6736910000fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.687361 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:06.687563 5513 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:06.687721 5513 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:06.689167 5280 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [term 1 FOLLOWER]: Refusing update from remote peer c1442b54fbc3492996df6736910000fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:06.689163 5355 raft_consensus.cc:1261] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [term 1 FOLLOWER]: Refusing update from remote peer c1442b54fbc3492996df6736910000fa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:06.689455 5463 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:06.689545 5511 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [LEADER]: Connected to new peer: Peer: permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:06.690824 5484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1442b54fbc3492996df6736910000fa. Latest consensus state: current_term: 1 leader_uuid: "c1442b54fbc3492996df6736910000fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.690827 5498 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1442b54fbc3492996df6736910000fa. Latest consensus state: current_term: 1 leader_uuid: "c1442b54fbc3492996df6736910000fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.690914 5484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:44:06.690932 5498 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:44:06.691996 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1442b54fbc3492996df6736910000fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.692083 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:06.692035 5498 sys_catalog.cc:455] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1442b54fbc3492996df6736910000fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.692121 5498 sys_catalog.cc:458] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:44:06.692183 5511 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c1442b54fbc3492996df6736910000fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.692332 5511 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:06.692752 5513 catalog_manager.cc:1347] Generated new cluster ID: 15e6f157f5db4fab906ce489746b1ae5
I20260926 05:44:06.692807 5513 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:44:06.697394 5484 sys_catalog.cc:455] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c1442b54fbc3492996df6736910000fa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b20a6589c89c41ce887898bdc9bca165" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 39273 } } peers { permanent_uuid: "e6ba4f1654814b4595f28acd5bb7f1fe" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33439 } } peers { permanent_uuid: "c1442b54fbc3492996df6736910000fa" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35513 } } }
I20260926 05:44:06.697476 5484 sys_catalog.cc:458] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:44:06.731566 5513 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:44:06.732244 5513 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:06.743227 5513 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c1442b54fbc3492996df6736910000fa: Generated new TSK 0
I20260926 05:44:06.743486 5513 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:44:07.377499 5495 catalog_manager.cc:1259] Loaded cluster ID: 15e6f157f5db4fab906ce489746b1ae5
I20260926 05:44:07.377588 5495 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: loading cluster ID for follower catalog manager: success
I20260926 05:44:07.378509 5495 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: acquiring CA information for follower catalog manager: success
I20260926 05:44:07.378851 5495 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b20a6589c89c41ce887898bdc9bca165: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:44:07.425920 5509 catalog_manager.cc:1259] Loaded cluster ID: 15e6f157f5db4fab906ce489746b1ae5
I20260926 05:44:07.425995 5509 catalog_manager.cc:1552] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: loading cluster ID for follower catalog manager: success
I20260926 05:44:07.426867 5509 catalog_manager.cc:1574] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: acquiring CA information for follower catalog manager: success
I20260926 05:44:07.427206 5509 catalog_manager.cc:1602] T 00000000000000000000000000000000 P e6ba4f1654814b4595f28acd5bb7f1fe: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:44:07.818910 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5238
I20260926 05:44:07.824298 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5313
I20260926 05:44:07.829129 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5388
2026-09-26T05:44:07Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3100 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:44:07 dist-test-slave-1fdt krb5kdc[5528](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:44:07 dist-test-slave-1fdt krb5kdc[5528](info): set up 2 sockets
Sep 26 05:44:07 dist-test-slave-1fdt krb5kdc[5528](info): commencing operation
krb5kdc: starting...
W20260926 05:44:08.881897 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:44:08 dist-test-slave-1fdt krb5kdc[5528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:44:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:44:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:44:09.005296 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43991
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43473
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:43991
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:09.085120 5541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:09.085378 5541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:09.085417 5541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:09.086581 5541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:09.086663 5541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:09.086687 5541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:09.086717 5541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:09.086746 5541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:09.088203 5541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43473
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:43991
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43991
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5541
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:09.088485 5541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:09.088728 5541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:09.093961 5549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:09.093987 5546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:09.094087 5541 server_base.cc:1034] running on GCE node
W20260926 05:44:09.093962 5547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:09.094545 5541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:09.094861 5541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:09.096035 5541 hybrid_clock.cc:648] HybridClock initialized: now 1790401449096001 us; error 53 us; skew 500 ppm
Sep 26 05:44:09 dist-test-slave-1fdt krb5kdc[5528](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401449, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:09.098687 5541 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:09.099186 5541 webserver.cc:460] Webserver started at http://127.23.213.254:35621/ using document root <none> and password file <none>
I20260926 05:44:09.099435 5541 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:09.099507 5541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:09.099658 5541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:09.100700 5541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "0a9c1240390e432487f157e30050ec10"
format_stamp: "Formatted at 2026-09-26 05:44:09 on dist-test-slave-1fdt"
server_key: "6d7db68be3033b961c3bd1a5acead678"
server_key_iv: "b41800d9e70f56d237693f559ac9472a"
server_key_version: "encryptionkey@0"
I20260926 05:44:09.101050 5541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "0a9c1240390e432487f157e30050ec10"
format_stamp: "Formatted at 2026-09-26 05:44:09 on dist-test-slave-1fdt"
server_key: "6d7db68be3033b961c3bd1a5acead678"
server_key_iv: "b41800d9e70f56d237693f559ac9472a"
server_key_version: "encryptionkey@0"
I20260926 05:44:09.102463 5541 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:44:09.103318 5556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:09.103538 5541 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:09.103660 5541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "0a9c1240390e432487f157e30050ec10"
format_stamp: "Formatted at 2026-09-26 05:44:09 on dist-test-slave-1fdt"
server_key: "6d7db68be3033b961c3bd1a5acead678"
server_key_iv: "b41800d9e70f56d237693f559ac9472a"
server_key_version: "encryptionkey@0"
I20260926 05:44:09.103745 5541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:09.115522 5541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:09.117419 5541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:09.117592 5541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:09.122280 5541 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:43991
I20260926 05:44:09.122324 5608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:43991 every 8 connection(s)
I20260926 05:44:09.122709 5541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:09.123729 5609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:09.126366 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10: Bootstrap starting.
I20260926 05:44:09.127022 5609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:09.127352 5609 log.cc:824] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:09.128105 5609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10: No bootstrap required, opened a new log
I20260926 05:44:09.129947 5609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9c1240390e432487f157e30050ec10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43991 } }
I20260926 05:44:09.130095 5609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:09.130118 5609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a9c1240390e432487f157e30050ec10, State: Initialized, Role: FOLLOWER
I20260926 05:44:09.130371 5609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9c1240390e432487f157e30050ec10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43991 } }
I20260926 05:44:09.130427 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5541
I20260926 05:44:09.130452 5609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:44:09.130515 5609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:44:09.130553 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:44:09.130570 5609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:09.130738 24407 external_mini_cluster.cc:1428] Setting key 47579ca1c92911bc3611fb8f86c0fc52
I20260926 05:44:09.131280 5609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9c1240390e432487f157e30050ec10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43991 } }
I20260926 05:44:09.131482 5609 leader_election.cc:302] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0a9c1240390e432487f157e30050ec10; no voters:
I20260926 05:44:09.131677 5609 leader_election.cc:288] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:44:09.131726 5614 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:09.131928 5614 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [term 1 LEADER]: Becoming Leader. State: Replica: 0a9c1240390e432487f157e30050ec10, State: Running, Role: LEADER
I20260926 05:44:09.131990 5609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:09.132119 5614 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9c1240390e432487f157e30050ec10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43991 } }
I20260926 05:44:09.133570 5615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0a9c1240390e432487f157e30050ec10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9c1240390e432487f157e30050ec10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43991 } } }
I20260926 05:44:09.133683 5615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:09.133653 5616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0a9c1240390e432487f157e30050ec10. Latest consensus state: current_term: 1 leader_uuid: "0a9c1240390e432487f157e30050ec10" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0a9c1240390e432487f157e30050ec10" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43991 } } }
I20260926 05:44:09.133773 5616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10 [sys.catalog]: This master's current role is: LEADER
W20260926 05:44:09.134601 5628 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:44:09.134661 5628 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:44:09.134740 5629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:09.134955 5629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:09.136587 5629 catalog_manager.cc:1347] Generated new cluster ID: 5712090b9bde4e7ebe6b805307251dde
I20260926 05:44:09.136636 5629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:44:09 dist-test-slave-1fdt krb5kdc[5528](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401448, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:09.143404 5613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:09.131222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56866 (local address 127.23.213.254:43991)
0926 05:44:09.131412 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:09.131423 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:09.131458 (+ 35us) server_negotiation.cc:407] Connection header received
0926 05:44:09.131599 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:09.131606 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:09.131673 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:09.132384 (+ 711us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:09.132659 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:09.134450 (+ 1791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:09.135554 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:09.136205 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:09.136346 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:09.139350 (+ 3004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:09.139354 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:09.139355 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:09.139366 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:09.141442 (+ 2076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:09.141943 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:09.141947 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:09.141948 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:09.142093 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:09.142338 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:09.142339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:09.142340 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:09.142715 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:09.142901 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:09.142982 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 05:44:09.143168 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":103,"threads_started":1}
I20260926 05:44:09.144209 5629 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:44:09.144387 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:44:09.144879 5629 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:09.151522 5629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0a9c1240390e432487f157e30050ec10: Generated new TSK 0
I20260926 05:44:09.151706 5629 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:44:09.198190 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5541
2026-09-26T05:44:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1367 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:44:09 dist-test-slave-1fdt krb5kdc[5636](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:44:09 dist-test-slave-1fdt krb5kdc[5636](info): set up 2 sockets
Sep 26 05:44:09 dist-test-slave-1fdt krb5kdc[5636](info): commencing operation
krb5kdc: starting...
W20260926 05:44:10.225991 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:44:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:44:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:44:10.345602 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41399
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:10.428475 5649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:10.428684 5649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:10.428706 5649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:10.429906 5649 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:10.429949 5649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:10.429965 5649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:10.429977 5649 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:10.429989 5649 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:10.431414 5649 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5649
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:10.431608 5649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:10.431774 5649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:10.436923 5655 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:10.436962 5654 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:10.436985 5649 server_base.cc:1034] running on GCE node
W20260926 05:44:10.436967 5657 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:10.437407 5649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:10.437676 5649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:10.438923 5649 hybrid_clock.cc:648] HybridClock initialized: now 1790401450438905 us; error 37 us; skew 500 ppm
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:10.441690 5649 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:10.442245 5649 webserver.cc:460] Webserver started at http://127.23.213.254:43717/ using document root <none> and password file <none>
I20260926 05:44:10.442448 5649 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:10.442494 5649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:10.442589 5649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:10.443677 5649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "9943514d9c374a8a83b4d597df63fbcd"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "d072bcf777ec714e00f60712af830c89"
server_key_iv: "4b66535e4ddbdaa26a85eb033100b681"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.444015 5649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "9943514d9c374a8a83b4d597df63fbcd"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "d072bcf777ec714e00f60712af830c89"
server_key_iv: "4b66535e4ddbdaa26a85eb033100b681"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.445406 5649 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:44:10.446242 5664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.446422 5649 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.446512 5649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "9943514d9c374a8a83b4d597df63fbcd"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "d072bcf777ec714e00f60712af830c89"
server_key_iv: "4b66535e4ddbdaa26a85eb033100b681"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.446568 5649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:10.455915 5649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:10.458094 5649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:10.458206 5649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:10.462023 5649 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:41399
I20260926 05:44:10.462139 5716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:41399 every 8 connection(s)
I20260926 05:44:10.462467 5649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:10.463617 5717 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:10.466240 5717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Bootstrap starting.
I20260926 05:44:10.466888 5717 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:10.467211 5717 log.cc:824] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:10.467936 5717 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: No bootstrap required, opened a new log
I20260926 05:44:10.469911 5717 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:10.470062 5717 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:10.470117 5717 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9943514d9c374a8a83b4d597df63fbcd, State: Initialized, Role: FOLLOWER
I20260926 05:44:10.470283 5717 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:10.470374 5717 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:44:10.470435 5717 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:44:10.470494 5717 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:10.471364 5717 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:10.471521 5717 leader_election.cc:302] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9943514d9c374a8a83b4d597df63fbcd; no voters:
I20260926 05:44:10.471735 5717 leader_election.cc:288] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:44:10.471733 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5649
I20260926 05:44:10.471762 5720 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:10.471849 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:44:10.471886 5720 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 LEADER]: Becoming Leader. State: Replica: 9943514d9c374a8a83b4d597df63fbcd, State: Running, Role: LEADER
I20260926 05:44:10.471976 5720 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:10.472014 5717 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:10.472054 24407 external_mini_cluster.cc:1428] Setting key fa5896dd5dc65b642adc2d3885a926a3
I20260926 05:44:10.472321 5721 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9943514d9c374a8a83b4d597df63fbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } } }
I20260926 05:44:10.472417 5721 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:10.472311 5720 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9943514d9c374a8a83b4d597df63fbcd. Latest consensus state: current_term: 1 leader_uuid: "9943514d9c374a8a83b4d597df63fbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } } }
I20260926 05:44:10.472641 5720 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:10.473263 5726 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:10.473829 5726 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:10.476426 5726 catalog_manager.cc:1347] Generated new cluster ID: 4b055c6fd1984ceab84ae4f50bd7b864
I20260926 05:44:10.476477 5726 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:10.485050 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.472697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.23.213.254:41399)
0926 05:44:10.472870 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:10.472882 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:10.473170 (+ 288us) server_negotiation.cc:407] Connection header received
0926 05:44:10.473328 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:10.473338 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:10.473408 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:10.473585 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:10.474408 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.475903 (+ 1495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.476900 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.477642 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.478330 (+ 688us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.480976 (+ 2646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:10.480981 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:10.480981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:10.480996 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:10.483278 (+ 2282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.483692 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.483696 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.483697 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.483827 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.484035 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.484036 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.484037 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.484386 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:10.484565 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:10.484641 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 05:44:10.484790 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":72,"threads_started":1}
I20260926 05:44:10.490777 5726 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:44:10.491546 5726 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:10.495213 5726 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Generated new TSK 0
I20260926 05:44:10.495386 5726 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:44:10.526772 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41399
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:10.618070 5743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:10.618311 5743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:10.618355 5743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:10.619537 5743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:10.619611 5743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:10.619659 5743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:44:10.621147 5743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:10.621428 5743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:10.621662 5743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:10.627719 5751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:10.627727 5748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:10.627758 5749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:10.628048 5743 server_base.cc:1034] running on GCE node
I20260926 05:44:10.628261 5743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:10.628521 5743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:10.629688 5743 hybrid_clock.cc:648] HybridClock initialized: now 1790401450629677 us; error 28 us; skew 500 ppm
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:10.632187 5743 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:44:10.632678 5743 webserver.cc:460] Webserver started at http://127.23.213.193:46291/ using document root <none> and password file <none>
I20260926 05:44:10.632915 5743 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:10.632994 5743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:10.633148 5743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:10.634294 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "638bbb3b54744426a64392bcb52d3387"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "b0785cd38a41711de3b224c8245d4790"
server_key_iv: "6f948e9c5a0581225a6092cb07f5d207"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.634657 5743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "638bbb3b54744426a64392bcb52d3387"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "b0785cd38a41711de3b224c8245d4790"
server_key_iv: "6f948e9c5a0581225a6092cb07f5d207"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.635993 5743 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:44:10.636744 5758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.636935 5743 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:10.637043 5743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "638bbb3b54744426a64392bcb52d3387"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "b0785cd38a41711de3b224c8245d4790"
server_key_iv: "6f948e9c5a0581225a6092cb07f5d207"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.637126 5743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:10.661847 5743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:10.664152 5743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:10.664335 5743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:10.664598 5743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:10.664939 5743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:10.665000 5743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.665041 5743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:10.665071 5743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.670974 5743 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:44987
I20260926 05:44:10.671007 5871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:44987 every 8 connection(s)
I20260926 05:44:10.671379 5743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:44:10.672663 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5743
I20260926 05:44:10.672782 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:44:10.673019 24407 external_mini_cluster.cc:1428] Setting key 9a5276f9a06b5b37c9980ee20e776dba
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
I20260926 05:44:10.681896 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.672118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:35515 (local address 127.23.213.254:41399)
0926 05:44:10.672210 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:10.672212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:10.672380 (+ 168us) server_negotiation.cc:407] Connection header received
0926 05:44:10.673156 (+ 776us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:10.673157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:10.673174 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:10.673228 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:10.674280 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.674871 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.675848 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.676182 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.676258 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.678808 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:10.678810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:10.678811 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:10.678820 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:10.680468 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.680865 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.680866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.680867 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.680933 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.681178 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.681179 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.681180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.681341 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:10.681413 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:10.681671 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 05:44:10.681780 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 05:44:10.682065 5874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.672064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:35515)
0926 05:44:10.672253 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:10.672295 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:10.673030 (+ 735us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:10.673296 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:10.673306 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:10.673977 (+ 671us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:10.674176 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:10.674180 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.674970 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.674971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:10.675783 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:10.675785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.676274 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.676274 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:10.676379 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.676990 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:10.676999 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:10.678705 (+ 1706us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:10.680555 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:10.680561 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:10.680563 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:10.680779 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:10.681006 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:10.681006 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:10.681006 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:10.681103 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:10.681408 (+ 305us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:10.681413 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:10.681536 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 05:44:10.681619 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:44:10.681816 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":104,"threads_started":1}
I20260926 05:44:10.682607 5872 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41399
I20260926 05:44:10.682699 5872 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:10.682933 5872 heartbeater.cc:511] Master 127.23.213.254:41399 requested a full tablet report, sending...
I20260926 05:44:10.683519 5681 ts_manager.cc:194] Registered new tserver with Master: 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987)
I20260926 05:44:10.684144 5681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:35515
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:44:10.700436 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41399
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:10.782100 5877 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:10.782315 5877 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:10.782338 5877 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:10.783794 5877 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:10.783852 5877 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:10.783885 5877 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:44:10.785844 5877 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5877
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:10.786104 5877 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:10.786355 5877 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:10.791684 5882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:10.791683 5883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:10.791715 5885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:10.791941 5877 server_base.cc:1034] running on GCE node
I20260926 05:44:10.792133 5877 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:10.792407 5877 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:10.793576 5877 hybrid_clock.cc:648] HybridClock initialized: now 1790401450793573 us; error 38 us; skew 500 ppm
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:10.796234 5877 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:44:10.796770 5877 webserver.cc:460] Webserver started at http://127.23.213.194:42703/ using document root <none> and password file <none>
I20260926 05:44:10.797050 5877 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:10.797123 5877 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:10.797281 5877 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:10.798439 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "ee15fd8ee555439cade240eddd614b42"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "4511c0ca2cc16629e9c66ae1e249a4cb"
server_key_iv: "6358f2b48c4fd41113fe50ad62e0063f"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.798822 5877 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "ee15fd8ee555439cade240eddd614b42"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "4511c0ca2cc16629e9c66ae1e249a4cb"
server_key_iv: "6358f2b48c4fd41113fe50ad62e0063f"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.800172 5877 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:44:10.800956 5892 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.801143 5877 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:10.801245 5877 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "ee15fd8ee555439cade240eddd614b42"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "4511c0ca2cc16629e9c66ae1e249a4cb"
server_key_iv: "6358f2b48c4fd41113fe50ad62e0063f"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.801328 5877 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:10.812894 5877 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:10.815692 5877 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:10.815815 5877 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:10.816128 5877 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:10.816457 5877 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:10.816491 5877 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.816550 5877 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:10.816578 5877 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.822430 5877 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:37827
I20260926 05:44:10.822577 6005 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:37827 every 8 connection(s)
I20260926 05:44:10.822865 5877 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:44:10.825379 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 5877
I20260926 05:44:10.825515 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:44:10.825762 24407 external_mini_cluster.cc:1428] Setting key 6f3beae006eb4c03c3ec40cbc8638ee1
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:10.833452 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.823719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:44203 (local address 127.23.213.254:41399)
0926 05:44:10.823791 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:10.823793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:10.823994 (+ 201us) server_negotiation.cc:407] Connection header received
0926 05:44:10.824470 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:10.824471 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:10.824481 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:10.824545 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:10.825584 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.826231 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.827339 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.827673 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.827753 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.830338 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:10.830340 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:10.830340 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:10.830351 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:10.831800 (+ 1449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.832236 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.832238 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.832238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.832301 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.832588 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.832589 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.832589 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.832706 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:10.832785 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:10.833203 (+ 418us) server_negotiation.cc:299] Negotiation successful
0926 05:44:10.833311 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:44:10.833491 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.823663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:44203)
0926 05:44:10.823846 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:10.823891 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:10.824385 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:10.824629 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:10.824633 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:10.825240 (+ 607us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:10.825442 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:10.825446 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.826329 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.826330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:10.827241 (+ 911us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:10.827244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.827777 (+ 533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.827779 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:10.827948 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.828509 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:10.828520 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:10.830233 (+ 1713us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:10.831900 (+ 1667us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:10.831907 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:10.831910 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:10.832152 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:10.832384 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:10.832385 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:10.832385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:10.832501 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:10.832789 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:10.832796 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:10.833003 (+ 207us) client_negotiation.cc:769] Sending connection context
0926 05:44:10.833141 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 05:44:10.833318 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":106,"threads_started":1}
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
I20260926 05:44:10.834141 6006 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41399
I20260926 05:44:10.834259 6006 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:10.834482 6006 heartbeater.cc:511] Master 127.23.213.254:41399 requested a full tablet report, sending...
I20260926 05:44:10.834920 5681 ts_manager.cc:194] Registered new tserver with Master: ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827)
I20260926 05:44:10.835443 5681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:44203
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:44:10.853885 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41399
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:10.936198 6011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:10.936474 6011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:10.936517 6011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:10.937726 6011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:10.937839 6011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:10.937882 6011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:44:10.939469 6011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:10.939754 6011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:10.939993 6011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:10.945304 6017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:10.945392 6011 server_base.cc:1034] running on GCE node
W20260926 05:44:10.945300 6016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:10.945340 6019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:10.945698 6011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:10.945981 6011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:10.947126 6011 hybrid_clock.cc:648] HybridClock initialized: now 1790401450947102 us; error 50 us; skew 500 ppm
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:10.949725 6011 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:44:10.950264 6011 webserver.cc:460] Webserver started at http://127.23.213.195:34199/ using document root <none> and password file <none>
I20260926 05:44:10.950496 6011 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:10.950577 6011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:10.950743 6011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:10.951825 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "38a89f69f706405590660e7e0f53b389"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "419d39b96fd577e6fb642e28f5ebe858"
server_key_iv: "0f6067ce899fb04d784f5db34da9d0e5"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.952198 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "38a89f69f706405590660e7e0f53b389"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "419d39b96fd577e6fb642e28f5ebe858"
server_key_iv: "0f6067ce899fb04d784f5db34da9d0e5"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.953538 6011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:44:10.954389 6026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.954617 6011 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:44:10.954720 6011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "38a89f69f706405590660e7e0f53b389"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "419d39b96fd577e6fb642e28f5ebe858"
server_key_iv: "0f6067ce899fb04d784f5db34da9d0e5"
server_key_version: "encryptionkey@0"
I20260926 05:44:10.954811 6011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:10.963536 6011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:10.965471 6011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:10.965662 6011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:10.965934 6011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:10.966248 6011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:10.966310 6011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.966351 6011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:10.966380 6011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:10.973043 6011 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:45139
I20260926 05:44:10.973140 6139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:45139 every 8 connection(s)
I20260926 05:44:10.973459 6011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:44:10.978988 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6011
I20260926 05:44:10.979096 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:44:10.979282 24407 external_mini_cluster.cc:1428] Setting key 6bb7139345ff5dccd14e0402dfc1c272
Sep 26 05:44:10 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401450, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:10.983000 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.974283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:45331 (local address 127.23.213.254:41399)
0926 05:44:10.974371 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:10.974373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:10.974697 (+ 324us) server_negotiation.cc:407] Connection header received
0926 05:44:10.975182 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:10.975183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:10.975194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:10.975266 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:10.976178 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.976735 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.977708 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.978017 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.978073 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.980596 (+ 2523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:10.980597 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:10.980597 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:10.980603 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:10.981817 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.982205 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.982205 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.982206 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.982256 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.982479 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.982479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.982480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.982580 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:10.982629 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:10.982876 (+ 247us) server_negotiation.cc:299] Negotiation successful
0926 05:44:10.982938 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 05:44:10.983237 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.974248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:45331)
0926 05:44:10.974418 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:10.974460 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:10.975078 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:10.975323 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:10.975327 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:10.975889 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:10.976091 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:10.976096 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.976821 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.976821 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:10.977628 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:10.977629 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.978104 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.978105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:10.978210 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.978686 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:10.978695 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:10.980514 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:10.981907 (+ 1393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:10.981913 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:10.981915 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:10.982132 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:10.982317 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:10.982317 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:10.982317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:10.982420 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:10.982637 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:10.982642 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:10.982760 (+ 118us) client_negotiation.cc:769] Sending connection context
0926 05:44:10.982844 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 05:44:10.982964 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":95,"threads_started":1}
I20260926 05:44:10.983829 6140 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41399
I20260926 05:44:10.983966 6140 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:10.984148 6140 heartbeater.cc:511] Master 127.23.213.254:41399 requested a full tablet report, sending...
I20260926 05:44:10.984580 5681 ts_manager.cc:194] Registered new tserver with Master: 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139)
I20260926 05:44:10.984925 5681 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:45331
I20260926 05:44:10.991139 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:44:10.998546 5731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:10.992480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38456 (local address 127.23.213.254:41399)
0926 05:44:10.992546 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:10.992548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:10.992602 (+ 54us) server_negotiation.cc:407] Connection header received
0926 05:44:10.992655 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:10.992655 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:10.992664 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:10.992713 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:10.993272 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.993898 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.994592 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:10.994773 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:10.994826 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:10.996078 (+ 1252us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:10.996079 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:10.996079 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:10.996086 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:10.997574 (+ 1488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.997928 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.997929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.997930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.997982 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:10.998134 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:10.998134 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:10.998134 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:10.998266 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:10.998334 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:10.998424 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 05:44:10.998481 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:44:11.000269 5681 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:44:11.001907 5681 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:44:11.008795 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.004404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:37827 (local address 127.0.0.1:50536)
0926 05:44:11.004782 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.004824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.004892 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.005231 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.005234 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.005236 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.005328 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.005331 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.006359 (+ 1028us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.006360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.007589 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.007591 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.008488 (+ 897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.008489 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.008583 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.008600 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.008666 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.008717 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":3,"thread_start_us":83,"threads_started":1}
I20260926 05:44:11.008898 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.004751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36736 (local address 127.23.213.193:44987)
0926 05:44:11.005176 (+ 425us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.005180 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.005189 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:44:11.005224 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.005226 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.005241 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.005351 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.005462 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.006573 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.007826 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.008521 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.008585 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.008633 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.008774 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.008814 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":81,"threads_started":1}
I20260926 05:44:11.008898 6152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.004652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44987 (local address 127.0.0.1:36736)
0926 05:44:11.004928 (+ 276us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.004938 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.005050 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.005348 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.005348 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.005350 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.005385 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.005387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.006687 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.006688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.007736 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.007738 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.008590 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.008591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.008661 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.008666 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.008757 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.008803 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":76,"threads_started":1}
I20260926 05:44:11.009202 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.004526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50536 (local address 127.23.213.194:37827)
0926 05:44:11.004659 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.004664 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.004985 (+ 321us) server_negotiation.cc:407] Connection header received
0926 05:44:11.005030 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.005033 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.005050 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.005138 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.005430 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.006234 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.007671 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.008391 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.008471 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.008529 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.009083 (+ 554us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.009127 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":65,"threads_started":1}
I20260926 05:44:11.009663 5806 tablet_service.cc:1467] Processing CreateTablet for tablet fe9a465845684dfd9a0c0c6938814a8a (DEFAULT_TABLE table=test-table [id=f2d56564874d4e5fb54fef51bd595f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:11.009981 5940 tablet_service.cc:1467] Processing CreateTablet for tablet fe9a465845684dfd9a0c0c6938814a8a (DEFAULT_TABLE table=test-table [id=f2d56564874d4e5fb54fef51bd595f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:11.010104 5806 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9a465845684dfd9a0c0c6938814a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:11.010319 5940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9a465845684dfd9a0c0c6938814a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:11.010319 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.004432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45139 (local address 127.0.0.1:42890)
0926 05:44:11.004782 (+ 350us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.004824 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.004892 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.005274 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.005275 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.005278 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.005339 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.005342 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.007813 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.007814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.008498 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.008500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.010083 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.010084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.010147 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.010151 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.010202 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.010241 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"thread_start_us":86,"threads_started":1}
I20260926 05:44:11.010617 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.004690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42890 (local address 127.23.213.195:45139)
0926 05:44:11.004867 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.004871 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.004883 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:44:11.004988 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.004991 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.005012 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.005091 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.005416 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.006579 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.008598 (+ 2019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.009921 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.009993 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.010046 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.010492 (+ 446us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.010539 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":76,"threads_started":1}
I20260926 05:44:11.011248 6074 tablet_service.cc:1467] Processing CreateTablet for tablet fe9a465845684dfd9a0c0c6938814a8a (DEFAULT_TABLE table=test-table [id=f2d56564874d4e5fb54fef51bd595f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:11.011550 6074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9a465845684dfd9a0c0c6938814a8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:11.012940 6157 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Bootstrap starting.
I20260926 05:44:11.013487 6158 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Bootstrap starting.
I20260926 05:44:11.013597 6157 tablet_bootstrap.cc:654] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:11.014210 6158 tablet_bootstrap.cc:654] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:11.014240 6157 log.cc:824] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:11.014536 6159 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Bootstrap starting.
I20260926 05:44:11.014760 6158 log.cc:824] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:11.015166 6159 tablet_bootstrap.cc:654] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:11.015192 6157 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: No bootstrap required, opened a new log
I20260926 05:44:11.015281 6157 ts_tablet_manager.cc:1397] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:44:11.015574 6158 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: No bootstrap required, opened a new log
I20260926 05:44:11.015658 6158 ts_tablet_manager.cc:1397] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:44:11.015754 6159 log.cc:824] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:11.016538 6159 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: No bootstrap required, opened a new log
I20260926 05:44:11.016618 6159 ts_tablet_manager.cc:1397] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:44:11.017033 6157 raft_consensus.cc:348] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.017200 6157 raft_consensus.cc:374] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:11.017243 6157 raft_consensus.cc:729] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee15fd8ee555439cade240eddd614b42, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.017349 6157 consensus_queue.cc:260] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.017620 6157 ts_tablet_manager.cc:1428] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:44:11.017638 6158 raft_consensus.cc:348] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.017779 6006 heartbeater.cc:503] Master 127.23.213.254:41399 was elected leader, sending a full tablet report...
I20260926 05:44:11.017804 6158 raft_consensus.cc:374] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:11.017843 6158 raft_consensus.cc:729] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638bbb3b54744426a64392bcb52d3387, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.018177 6158 consensus_queue.cc:260] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.018492 5872 heartbeater.cc:503] Master 127.23.213.254:41399 was elected leader, sending a full tablet report...
I20260926 05:44:11.018496 6158 ts_tablet_manager.cc:1428] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 05:44:11.018657 6159 raft_consensus.cc:348] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.018795 6159 raft_consensus.cc:374] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:11.018839 6159 raft_consensus.cc:729] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a89f69f706405590660e7e0f53b389, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.018965 6159 consensus_queue.cc:260] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.019232 6159 ts_tablet_manager.cc:1428] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:44:11.019260 6140 heartbeater.cc:503] Master 127.23.213.254:41399 was elected leader, sending a full tablet report...
I20260926 05:44:11.022048 6163 raft_consensus.cc:482] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:11.022126 6163 raft_consensus.cc:504] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.022535 6163 leader_election.cc:288] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987)
I20260926 05:44:11.025661 6008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.022719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45139 (local address 127.23.213.194:50169)
0926 05:44:11.022777 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.022789 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.022831 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.023032 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.023034 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.023036 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.023123 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.023125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.024091 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.024091 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.024837 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.024840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.025368 (+ 528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.025369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.025451 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.025454 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.025545 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.025590 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:44:11.025660 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.022791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:50169 (local address 127.23.213.195:45139)
0926 05:44:11.022844 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.022846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.022856 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:44:11.022894 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.022894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.022902 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.022981 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.023199 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.024005 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.024918 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.025272 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.025370 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.025379 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.025554 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.025599 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:44:11.026026 6094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9a465845684dfd9a0c0c6938814a8a" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a89f69f706405590660e7e0f53b389" is_pre_election: true
I20260926 05:44:11.026199 6094 raft_consensus.cc:2454] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 0.
I20260926 05:44:11.026412 5896 leader_election.cc:302] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a89f69f706405590660e7e0f53b389, ee15fd8ee555439cade240eddd614b42; no voters:
I20260926 05:44:11.026542 6163 raft_consensus.cc:2781] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:44:11.026589 6163 raft_consensus.cc:482] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:44:11.026625 6163 raft_consensus.cc:3037] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:11.026634 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.022906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:47565 (local address 127.23.213.193:44987)
0926 05:44:11.022971 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.022973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.023010 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:44:11.023050 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.023051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.023060 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.023147 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.023267 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.024005 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.025716 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.026232 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.026302 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.026313 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.026529 (+ 216us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.026564 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:44:11.026798 6166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.022727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44987 (local address 127.23.213.194:47565)
0926 05:44:11.022969 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.022977 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.023014 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.023149 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.023150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.023151 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.023204 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.023205 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.024922 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.024922 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.025602 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.025604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.026379 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.026380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.026442 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.026446 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.026524 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.026563 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":228,"mutex_wait_us":9,"thread_start_us":81,"threads_started":1}
I20260926 05:44:11.027132 5826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9a465845684dfd9a0c0c6938814a8a" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638bbb3b54744426a64392bcb52d3387" is_pre_election: true
I20260926 05:44:11.027240 5826 raft_consensus.cc:2454] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 0.
I20260926 05:44:11.027551 6163 raft_consensus.cc:504] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.027688 6163 leader_election.cc:288] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 election: Requested vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987)
I20260926 05:44:11.027808 5826 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9a465845684dfd9a0c0c6938814a8a" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638bbb3b54744426a64392bcb52d3387"
I20260926 05:44:11.027832 6094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9a465845684dfd9a0c0c6938814a8a" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a89f69f706405590660e7e0f53b389"
I20260926 05:44:11.027868 5826 raft_consensus.cc:3037] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:11.027889 6094 raft_consensus.cc:3037] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:11.028705 5826 raft_consensus.cc:2454] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 1.
I20260926 05:44:11.028705 6094 raft_consensus.cc:2454] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 1.
I20260926 05:44:11.028852 5896 leader_election.cc:302] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a89f69f706405590660e7e0f53b389, ee15fd8ee555439cade240eddd614b42; no voters:
I20260926 05:44:11.028930 6163 raft_consensus.cc:2781] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:11.029057 6163 raft_consensus.cc:686] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 1 LEADER]: Becoming Leader. State: Replica: ee15fd8ee555439cade240eddd614b42, State: Running, Role: LEADER
I20260926 05:44:11.029162 6163 consensus_queue.cc:237] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.030052 5681 catalog_manager.cc:5495] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee15fd8ee555439cade240eddd614b42 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "ee15fd8ee555439cade240eddd614b42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } health_report { overall_health: UNKNOWN } } }
I20260926 05:44:11.038311 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.035982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50548 (local address 127.23.213.194:37827)
0926 05:44:11.036048 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.036050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.036059 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:44:11.036097 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.036098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.036121 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.036181 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:44:11.036315 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.036783 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.037489 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.037673 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.037725 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.037938 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:44:11.038039 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:44:11.038116 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.038156 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.038175 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:44:11.040200 6094 raft_consensus.cc:1261] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Refusing update from remote peer ee15fd8ee555439cade240eddd614b42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:11.040213 5826 raft_consensus.cc:1261] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Refusing update from remote peer ee15fd8ee555439cade240eddd614b42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:11.040649 6167 consensus_queue.cc:1035] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:11.040810 6163 consensus_queue.cc:1035] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:11.043224 6170 mvcc.cc:205] Tried to move back new op lower bound from 7333484343458107392 to 7333484343416205312. Current Snapshot: MvccSnapshot[applied={T|T < 7333484343458107392}]
I20260926 05:44:11.043390 6171 mvcc.cc:205] Tried to move back new op lower bound from 7333484343458107392 to 7333484343416205312. Current Snapshot: MvccSnapshot[applied={T|T < 7333484343458107392}]
I20260926 05:44:11.044132 6172 mvcc.cc:205] Tried to move back new op lower bound from 7333484343458107392 to 7333484343416205312. Current Snapshot: MvccSnapshot[applied={T|T < 7333484343458107392}]
I20260926 05:44:11.045374 5680 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:11.045490 5680 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:11.046281 5680 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Sending DeleteTablet for 3 replicas of tablet fe9a465845684dfd9a0c0c6938814a8a
I20260926 05:44:11.046648 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5743
I20260926 05:44:11.046721 6074 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9a465845684dfd9a0c0c6938814a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:11 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42890
I20260926 05:44:11.046725 5806 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9a465845684dfd9a0c0c6938814a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:11 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:36736
I20260926 05:44:11.047039 6179 tablet_replica.cc:330] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: stopping tablet replica
I20260926 05:44:11.047308 6179 raft_consensus.cc:2229] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:11.047297 5940 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9a465845684dfd9a0c0c6938814a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:11 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:50536
I20260926 05:44:11.047484 6179 raft_consensus.cc:2258] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:11.047974 6179 ts_tablet_manager.cc:1905] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:11.050096 6179 ts_tablet_manager.cc:1918] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:11.050220 6179 log.cc:1196] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/fe9a465845684dfd9a0c0c6938814a8a
I20260926 05:44:11.050606 6179 ts_tablet_manager.cc:1939] T fe9a465845684dfd9a0c0c6938814a8a P 38a89f69f706405590660e7e0f53b389: Deleting consensus metadata
I20260926 05:44:11.051095 5668 catalog_manager.cc:4841] TS 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139): tablet fe9a465845684dfd9a0c0c6938814a8a (table test-table [id=f2d56564874d4e5fb54fef51bd595f04]) successfully deleted
I20260926 05:44:11.051146 6180 tablet_replica.cc:330] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: stopping tablet replica
I20260926 05:44:11.051350 6180 raft_consensus.cc:2229] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:44:11.051499 6180 raft_consensus.cc:2258] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:11.051997 6180 ts_tablet_manager.cc:1905] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:11.054036 6180 ts_tablet_manager.cc:1918] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:11.054174 6180 log.cc:1196] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/fe9a465845684dfd9a0c0c6938814a8a
W20260926 05:44:11.054400 5667 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:44987 (error 108)
I20260926 05:44:11.054510 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5877
I20260926 05:44:11.054553 6180 ts_tablet_manager.cc:1939] T fe9a465845684dfd9a0c0c6938814a8a P ee15fd8ee555439cade240eddd614b42: Deleting consensus metadata
I20260926 05:44:11.055595 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.054887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44987 (local address 127.0.0.1:36750)
0926 05:44:11.055509 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.055554 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":584}
W20260926 05:44:11.055758 5667 catalog_manager.cc:4584] TS 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fe9a465845684dfd9a0c0c6938814a8a: Network error: Client connection negotiation failed: client connection to 127.23.213.193:44987: connect: Connection refused (error 111)
I20260926 05:44:11.059909 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 6011
I20260926 05:44:11.059995 6151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.059867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:37827 (local address 127.0.0.1:50562)
0926 05:44:11.059921 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.059960 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.194:37827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260926 05:44:11.064786 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 5649
I20260926 05:44:11.070143 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41399
--webserver_interface=127.23.213.254
--webserver_port=43717
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:11.152244 6182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:11.152452 6182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:11.152474 6182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:11.153566 6182 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:11.153600 6182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:11.153614 6182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:11.153626 6182 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:11.153640 6182 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:11.155048 6182 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=43717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:11.155246 6182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:11.155427 6182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:11.164439 6189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.164433 6191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.164433 6188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:11.164820 6182 server_base.cc:1034] running on GCE node
I20260926 05:44:11.165104 6182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:11.165453 6182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:11.166635 6182 hybrid_clock.cc:648] HybridClock initialized: now 1790401451166630 us; error 33 us; skew 500 ppm
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:11.169303 6182 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:11.169857 6182 webserver.cc:460] Webserver started at http://127.23.213.254:43717/ using document root <none> and password file <none>
I20260926 05:44:11.170073 6182 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:11.170125 6182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:11.171468 6182 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:44:11.172109 6198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.172289 6182 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:44:11.172375 6182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "9943514d9c374a8a83b4d597df63fbcd"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "d072bcf777ec714e00f60712af830c89"
server_key_iv: "4b66535e4ddbdaa26a85eb033100b681"
server_key_version: "encryptionkey@0"
I20260926 05:44:11.172775 6182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:11.213711 6182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:11.215862 6182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:11.215976 6182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:11.220053 6182 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:41399
I20260926 05:44:11.220144 6250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:41399 every 8 connection(s)
I20260926 05:44:11.220541 6182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:11.222259 6251 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:44:11.222945 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Bootstrap starting.
I20260926 05:44:11.225667 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6182
I20260926 05:44:11.225975 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:44987
--local_ip_for_outbound_sockets=127.23.213.193
--tserver_master_addrs=127.23.213.254:41399
--webserver_port=46291
--webserver_interface=127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:11.231482 6251 log.cc:824] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:11.233490 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:44:11.233891 6251 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Bootstrap complete.
I20260926 05:44:11.236210 6251 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:11.236474 6251 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9943514d9c374a8a83b4d597df63fbcd, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.236686 6251 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:11.236786 6251 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:44:11.236847 6251 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:44:11.236904 6251 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:44:11.238008 6251 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:11.238128 6251 leader_election.cc:302] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9943514d9c374a8a83b4d597df63fbcd; no voters:
I20260926 05:44:11.238334 6251 leader_election.cc:288] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:44:11.238372 6255 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:44:11.238502 6255 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 LEADER]: Becoming Leader. State: Replica: 9943514d9c374a8a83b4d597df63fbcd, State: Running, Role: LEADER
I20260926 05:44:11.238606 6251 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:11.238585 6255 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:11.238870 6255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9943514d9c374a8a83b4d597df63fbcd. Latest consensus state: current_term: 2 leader_uuid: "9943514d9c374a8a83b4d597df63fbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } } }
I20260926 05:44:11.238932 6255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:11.239094 6256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9943514d9c374a8a83b4d597df63fbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } } }
I20260926 05:44:11.239202 6256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:11.240685 6270 catalog_manager.cc:1259] Loaded cluster ID: 4b055c6fd1984ceab84ae4f50bd7b864
I20260926 05:44:11.240736 6270 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: loading cluster ID for follower catalog manager: success
W20260926 05:44:11.240943 6270 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: acquiring CA information for follower catalog manager: Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG
W20260926 05:44:11.240985 6270 catalog_manager.cc:873] Runtime error: could not decrypt private key with the password returned by the configured command: unable to load data from memory: error reading private key: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D07803A:asn1 encoding routines:ASN1_ITEM_EX_D2I:nested asn1 error:tasn_dec.c:386:Type=X509_ALGOR error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=algor, Type=X509_SIG: failed to prepare follower catalog manager, will retry
I20260926 05:44:11.241019 6259 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:11.241379 6259 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:11.241457 6259 catalog_manager.cc:1259] Loaded cluster ID: 4b055c6fd1984ceab84ae4f50bd7b864
I20260926 05:44:11.241478 6259 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:44:11.241624 6259 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 05:44:11.246337 6259 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 05:44:11.249583 6259 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:11.249841 6259 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Loaded TSK: 0
I20260926 05:44:11.250073 6259 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:44:11.309687 6253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:11.309973 6253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:11.310010 6253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:11.311185 6253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:11.311251 6253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:11.311295 6253 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:44:11.312685 6253 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:44987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=46291
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:11.312944 6253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:11.313180 6253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:11.318516 6281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.318531 6278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.318519 6279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:11.319114 6253 server_base.cc:1034] running on GCE node
I20260926 05:44:11.319335 6253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:11.319588 6253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:11.320747 6253 hybrid_clock.cc:648] HybridClock initialized: now 1790401451320719 us; error 45 us; skew 500 ppm
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:11.323439 6253 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:44:11.323948 6253 webserver.cc:460] Webserver started at http://127.23.213.193:46291/ using document root <none> and password file <none>
I20260926 05:44:11.324182 6253 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:11.324265 6253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:11.325681 6253 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:44:11.326447 6288 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.326642 6253 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:11.326754 6253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "638bbb3b54744426a64392bcb52d3387"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "b0785cd38a41711de3b224c8245d4790"
server_key_iv: "6f948e9c5a0581225a6092cb07f5d207"
server_key_version: "encryptionkey@0"
I20260926 05:44:11.327111 6253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:11.341567 6253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:11.343856 6253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:11.344031 6253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:11.344317 6253 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:11.344771 6295 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:44:11.346284 6253 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:44:11.346338 6253 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:44:11.346364 6253 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:44:11.347026 6253 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:44:11.347078 6253 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:11.347115 6295 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Bootstrap starting.
I20260926 05:44:11.354705 6253 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:44987
I20260926 05:44:11.355223 6253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:44:11.355618 6402 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:44987 every 8 connection(s)
I20260926 05:44:11.359122 6295 log.cc:824] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:11.361078 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6253
I20260926 05:44:11.361410 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:37827
--local_ip_for_outbound_sockets=127.23.213.194
--tserver_master_addrs=127.23.213.254:41399
--webserver_port=42703
--webserver_interface=127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:11.363008 6295 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:44:11.363381 6295 tablet_bootstrap.cc:492] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Bootstrap complete.
I20260926 05:44:11.364544 6295 ts_tablet_manager.cc:1397] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Time spent bootstrapping tablet: real 0.017s user 0.011s sys 0.003s
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:11.365852 6295 raft_consensus.cc:348] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.366235 6295 raft_consensus.cc:729] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638bbb3b54744426a64392bcb52d3387, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.366400 6295 consensus_queue.cc:260] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.366710 6295 ts_tablet_manager.cc:1428] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Time spent starting tablet: real 0.002s user 0.001s sys 0.003s
I20260926 05:44:11.369731 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.356104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:44563 (local address 127.23.213.254:41399)
0926 05:44:11.356308 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.356317 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.356356 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:44:11.357322 (+ 966us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.357330 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.357388 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.357539 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:11.358595 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.360003 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.361815 (+ 1812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.362484 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.363188 (+ 704us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.365472 (+ 2284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:11.365479 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:11.365479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:11.365496 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:11.367568 (+ 2072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.368076 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.368079 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.368080 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.368204 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.368524 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.368525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.368525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.368873 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:11.368979 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.369372 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.369526 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":125,"threads_started":1}
I20260926 05:44:11.369776 6405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.356033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:44563)
0926 05:44:11.356222 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.356266 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.357074 (+ 808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.357641 (+ 567us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.357646 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.358101 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:11.358447 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.358457 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.360139 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.360140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.361439 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.361442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.362625 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.362627 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.362802 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.363436 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:11.363447 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:11.365312 (+ 1865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:11.367658 (+ 2346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:11.367665 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:11.367667 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:11.367978 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:11.368292 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:11.368293 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:11.368294 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:11.368439 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:11.368964 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:11.368972 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:11.369162 (+ 190us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.369293 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.369487 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":118,"threads_started":1}
I20260926 05:44:11.370522 6403 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41399
I20260926 05:44:11.370622 6403 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:11.370785 6403 heartbeater.cc:511] Master 127.23.213.254:41399 requested a full tablet report, sending...
I20260926 05:44:11.371534 6215 ts_manager.cc:194] Registered new tserver with Master: 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987)
I20260926 05:44:11.372869 6215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:44563
I20260926 05:44:11.374020 6403 heartbeater.cc:503] Master 127.23.213.254:41399 was elected leader, sending a full tablet report...
I20260926 05:44:11.378216 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.372228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44987 (local address 127.0.0.1:36766)
0926 05:44:11.373888 (+ 1660us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.373930 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.374001 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.374411 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.374415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.374418 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.374493 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.374495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.375859 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.375860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.377029 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.377032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.377945 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.377946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.378034 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.378047 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.378121 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.378159 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1622,"thread_start_us":99,"threads_started":1}
I20260926 05:44:11.378265 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.372485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36766 (local address 127.23.213.193:44987)
0926 05:44:11.374229 (+ 1744us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.374233 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.374242 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:44:11.374279 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.374282 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.374298 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.374574 (+ 276us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.374623 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.375762 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.377120 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.377856 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.377932 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.377983 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.378155 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.378196 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1708,"thread_start_us":75,"threads_started":1}
I20260926 05:44:11.378655 6337 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9a465845684dfd9a0c0c6938814a8a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:11 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:36766
I20260926 05:44:11.379199 6412 tablet_replica.cc:330] stopping tablet replica
I20260926 05:44:11.379323 6412 raft_consensus.cc:2229] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:11.379420 6412 raft_consensus.cc:2258] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:11.379703 6412 ts_tablet_manager.cc:1905] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:11.381839 6412 ts_tablet_manager.cc:1918] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:11.381960 6412 log.cc:1196] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/fe9a465845684dfd9a0c0c6938814a8a
I20260926 05:44:11.382195 6412 ts_tablet_manager.cc:1939] T fe9a465845684dfd9a0c0c6938814a8a P 638bbb3b54744426a64392bcb52d3387: Deleting consensus metadata
I20260926 05:44:11.382736 6201 catalog_manager.cc:4841] TS 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987): tablet fe9a465845684dfd9a0c0c6938814a8a (table test-table [id=f2d56564874d4e5fb54fef51bd595f04]) successfully deleted
W20260926 05:44:11.450637 6408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:11.450892 6408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:11.450934 6408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:11.452087 6408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:11.452169 6408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:11.452217 6408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:44:11.453714 6408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:37827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=42703
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6408
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:11.454015 6408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:11.454262 6408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:11.459672 6418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.459705 6417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:11.459684 6408 server_base.cc:1034] running on GCE node
W20260926 05:44:11.459683 6420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:11.460204 6408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:11.460512 6408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:11.461678 6408 hybrid_clock.cc:648] HybridClock initialized: now 1790401451461662 us; error 31 us; skew 500 ppm
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:11.464327 6408 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:44:11.464815 6408 webserver.cc:460] Webserver started at http://127.23.213.194:42703/ using document root <none> and password file <none>
I20260926 05:44:11.465060 6408 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:11.465118 6408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:11.466692 6408 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:11.467298 6427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.467480 6408 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:11.467592 6408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "ee15fd8ee555439cade240eddd614b42"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "4511c0ca2cc16629e9c66ae1e249a4cb"
server_key_iv: "6358f2b48c4fd41113fe50ad62e0063f"
server_key_version: "encryptionkey@0"
I20260926 05:44:11.467962 6408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:11.503077 6408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:11.505192 6408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:11.505367 6408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:11.505625 6408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:11.505954 6408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:11.506013 6408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.506053 6408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:11.506078 6408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.512233 6408 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:37827
I20260926 05:44:11.512279 6540 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:37827 every 8 connection(s)
I20260926 05:44:11.512622 6408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:11.521087 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6408
I20260926 05:44:11.521483 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:45139
--local_ip_for_outbound_sockets=127.23.213.195
--tserver_master_addrs=127.23.213.254:41399
--webserver_port=34199
--webserver_interface=127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:11.522007 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.513433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:46605 (local address 127.23.213.254:41399)
0926 05:44:11.513559 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.513561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.513649 (+ 88us) server_negotiation.cc:407] Connection header received
0926 05:44:11.514230 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.514231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.514242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.514295 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:11.515214 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.515711 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.516678 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.516868 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.516918 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.519204 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:11.519205 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:11.519206 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:11.519212 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:11.520427 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.520799 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.520799 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.520800 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.520848 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.521183 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.521184 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.521184 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.521302 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:11.521353 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.521728 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.521848 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260926 05:44:11.522580 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.513360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:46605)
0926 05:44:11.513546 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.513584 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.514134 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.514350 (+ 216us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.514353 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.514929 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:11.515123 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.515129 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.515801 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.515802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.516605 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.516607 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.516955 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.516956 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.517056 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.517671 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:11.517680 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:11.519122 (+ 1442us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:11.520508 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:11.520514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:11.520515 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:11.520730 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:11.520935 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:11.520936 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:11.520936 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:11.521086 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:11.521370 (+ 284us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:11.521376 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:11.521548 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.522112 (+ 564us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.522303 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260926 05:44:11.523221 6541 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41399
I20260926 05:44:11.523334 6541 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:11.523548 6541 heartbeater.cc:511] Master 127.23.213.254:41399 requested a full tablet report, sending...
I20260926 05:44:11.523944 6215 ts_manager.cc:194] Registered new tserver with Master: ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827)
I20260926 05:44:11.524484 6215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:46605
W20260926 05:44:11.603770 6544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:11.603977 6544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:11.604000 6544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:11.605118 6544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:11.605150 6544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:11.605175 6544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:44:11.606719 6544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:45139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=34199
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:11.606910 6544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:11.607088 6544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:11.612154 6549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.612161 6552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.612164 6550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:11.612502 6544 server_base.cc:1034] running on GCE node
I20260926 05:44:11.612710 6544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:11.612964 6544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:11.614143 6544 hybrid_clock.cc:648] HybridClock initialized: now 1790401451614138 us; error 41 us; skew 500 ppm
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:11.616577 6544 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:44:11.617018 6544 webserver.cc:460] Webserver started at http://127.23.213.195:34199/ using document root <none> and password file <none>
I20260926 05:44:11.617262 6544 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:11.617355 6544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:11.618785 6544 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:11.619415 6559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.619585 6544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:11.619688 6544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "38a89f69f706405590660e7e0f53b389"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "419d39b96fd577e6fb642e28f5ebe858"
server_key_iv: "0f6067ce899fb04d784f5db34da9d0e5"
server_key_version: "encryptionkey@0"
I20260926 05:44:11.620040 6544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:11.634366 6544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:11.636368 6544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:11.636543 6544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:11.636790 6544 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:11.637104 6544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:11.637162 6544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.637204 6544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:11.637233 6544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.643656 6544 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:45139
I20260926 05:44:11.643729 6672 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:45139 every 8 connection(s)
I20260926 05:44:11.644109 6544 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:44:11.647821 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6544
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:11.654577 6406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.645008 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:38717 (local address 127.23.213.254:41399)
0926 05:44:11.645079 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.645081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.645232 (+ 151us) server_negotiation.cc:407] Connection header received
0926 05:44:11.645721 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.645722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.645734 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.645864 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:11.646814 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.647312 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.648563 (+ 1251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.648913 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.648995 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.651490 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:11.651491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:11.651491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:11.651498 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:11.653009 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.653458 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.653459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.653460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.653517 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.653819 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.653820 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.653820 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.653942 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:11.654025 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.654346 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.654455 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:44:11.654749 6675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.644908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:38717)
0926 05:44:11.645070 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.645112 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.645602 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.645920 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.645923 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.646471 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:11.646671 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.646676 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.647401 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.647402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.648456 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.648459 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.649003 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.649004 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.649104 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.649651 (+ 547us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:11.649661 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:11.651368 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:11.653113 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:11.653119 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:11.653122 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:11.653345 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:11.653596 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:11.653597 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:11.653597 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:11.653704 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:11.654039 (+ 335us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:11.654045 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:11.654177 (+ 132us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.654308 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.654483 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":93,"threads_started":1}
I20260926 05:44:11.655265 6673 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41399
I20260926 05:44:11.655331 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.648262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38458 (local address 127.23.213.254:41399)
0926 05:44:11.648380 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.648382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.648391 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:44:11.648452 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.648452 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.648464 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.648531 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:11.649225 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.649920 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.650912 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.651151 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.651205 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.652391 (+ 1186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:11.652392 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:11.652393 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:11.652399 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:11.653931 (+ 1532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.654257 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.654258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.654258 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.654321 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:11.654547 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:11.654548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:11.654548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:11.654673 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:11.654838 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.655185 (+ 347us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.655271 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":67,"threads_started":1}
I20260926 05:44:11.655368 6673 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:11.655558 6673 heartbeater.cc:511] Master 127.23.213.254:41399 requested a full tablet report, sending...
I20260926 05:44:11.656172 6215 ts_manager.cc:194] Registered new tserver with Master: 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139)
I20260926 05:44:11.656495 6215 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:38717
I20260926 05:44:11.667147 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:44:11.670184 6677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.667615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38466 (local address 127.23.213.254:41399)
0926 05:44:11.667672 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.667673 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.667768 (+ 95us) server_negotiation.cc:407] Connection header received
0926 05:44:11.667818 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.667819 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.667852 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.667920 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:44:11.668043 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.668524 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.669271 (+ 747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.669468 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.669523 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.669723 (+ 200us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:44:11.669922 (+ 199us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:44:11.670031 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.670103 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.670133 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:44:11.670487 6215 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:38466:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:44:11.672205 6215 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:44:11.674773 6337 tablet_service.cc:1467] Processing CreateTablet for tablet 3968e0011bc14a20b9e912ab30ac7a50 (DEFAULT_TABLE table=test-table [id=a73a3014dd5e4528a36aaacfd9876616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:11.674986 6337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3968e0011bc14a20b9e912ab30ac7a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:11.677060 6295 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Bootstrap starting.
I20260926 05:44:11.677381 6295 tablet_bootstrap.cc:654] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:11.678063 6295 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: No bootstrap required, opened a new log
I20260926 05:44:11.678115 6295 ts_tablet_manager.cc:1397] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:11.678171 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.674520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50570 (local address 127.23.213.194:37827)
0926 05:44:11.674687 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.674692 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.674834 (+ 142us) server_negotiation.cc:407] Connection header received
0926 05:44:11.674887 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.674890 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.674905 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.674976 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.675195 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.675963 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.677040 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.677818 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.677912 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.677975 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.678051 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.678087 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":95,"threads_started":1}
I20260926 05:44:11.678223 6295 raft_consensus.cc:348] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.678299 6295 raft_consensus.cc:374] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:11.678350 6295 raft_consensus.cc:729] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638bbb3b54744426a64392bcb52d3387, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.678401 6295 consensus_queue.cc:260] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.678457 6679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.674342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:37827 (local address 127.0.0.1:50570)
0926 05:44:11.674702 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.674712 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.674771 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.675077 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.675078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.675080 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.675122 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.675124 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.676072 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.676074 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.676903 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.676906 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.677910 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.677910 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.677972 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.677976 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.678038 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.678082 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":72,"threads_started":1}
I20260926 05:44:11.678514 6295 ts_tablet_manager.cc:1428] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.678782 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.674276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45139 (local address 127.0.0.1:42896)
0926 05:44:11.674318 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.674329 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.674369 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.674754 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.674755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.674756 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.674803 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.674805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.676523 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.676524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.677536 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.677539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.678566 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.678566 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.678627 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.678631 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.678680 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.678725 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 05:44:11.679031 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.674369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42896 (local address 127.23.213.195:45139)
0926 05:44:11.674531 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.674536 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.674553 (+ 17us) server_negotiation.cc:407] Connection header received
0926 05:44:11.674614 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.674616 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.674632 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.674752 (+ 120us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.674870 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.676357 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.677647 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.678449 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.678564 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.678622 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.678886 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.678937 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":92,"threads_started":1}
I20260926 05:44:11.679261 6475 tablet_service.cc:1467] Processing CreateTablet for tablet 3968e0011bc14a20b9e912ab30ac7a50 (DEFAULT_TABLE table=test-table [id=a73a3014dd5e4528a36aaacfd9876616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:11.679626 6475 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3968e0011bc14a20b9e912ab30ac7a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:11.679704 6607 tablet_service.cc:1467] Processing CreateTablet for tablet 3968e0011bc14a20b9e912ab30ac7a50 (DEFAULT_TABLE table=test-table [id=a73a3014dd5e4528a36aaacfd9876616]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:11.680018 6607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3968e0011bc14a20b9e912ab30ac7a50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:11.683739 6683 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Bootstrap starting.
I20260926 05:44:11.684365 6683 tablet_bootstrap.cc:654] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:11.684930 6683 log.cc:824] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:11.685225 6684 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Bootstrap starting.
I20260926 05:44:11.685674 6683 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: No bootstrap required, opened a new log
I20260926 05:44:11.685732 6683 ts_tablet_manager.cc:1397] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:44:11.685817 6684 tablet_bootstrap.cc:654] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:11.686249 6684 log.cc:824] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:11.686964 6684 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: No bootstrap required, opened a new log
I20260926 05:44:11.687050 6684 ts_tablet_manager.cc:1397] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:44:11.687911 6683 raft_consensus.cc:348] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.688040 6683 raft_consensus.cc:374] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:11.688060 6683 raft_consensus.cc:729] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee15fd8ee555439cade240eddd614b42, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.688189 6683 consensus_queue.cc:260] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.688493 6683 ts_tablet_manager.cc:1428] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:44:11.688500 6541 heartbeater.cc:503] Master 127.23.213.254:41399 was elected leader, sending a full tablet report...
I20260926 05:44:11.688854 6684 raft_consensus.cc:348] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.689025 6684 raft_consensus.cc:374] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:11.689080 6684 raft_consensus.cc:729] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38a89f69f706405590660e7e0f53b389, State: Initialized, Role: FOLLOWER
I20260926 05:44:11.689200 6684 consensus_queue.cc:260] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.689491 6684 ts_tablet_manager.cc:1428] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:44:11.689529 6673 heartbeater.cc:503] Master 127.23.213.254:41399 was elected leader, sending a full tablet report...
W20260926 05:44:11.763299 6542 tablet.cc:2369] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:44:11.815815 6687 raft_consensus.cc:482] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:11.816015 6687 raft_consensus.cc:504] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.816483 6687 leader_election.cc:288] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987)
I20260926 05:44:11.820431 6680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.816795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:54787 (local address 127.23.213.195:45139)
0926 05:44:11.816841 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.816843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.816855 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:44:11.816900 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.816901 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.816910 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.816975 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.817167 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.817960 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.819449 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.820021 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.820102 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.820111 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.820314 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.820356 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:44:11.820431 6411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.816758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:33459 (local address 127.23.213.193:44987)
0926 05:44:11.816822 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.816824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.817037 (+ 213us) server_negotiation.cc:407] Connection header received
0926 05:44:11.817099 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.817100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.817107 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.817184 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:11.817340 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.818039 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.819447 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.820021 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.820104 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.820111 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.820313 (+ 202us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.820356 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:44:11.820473 6543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.816675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45139 (local address 127.23.213.194:54787)
0926 05:44:11.816741 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.816754 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.816844 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.816994 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.816995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.816996 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.817078 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.817080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.818063 (+ 983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.818064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.819340 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.819344 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.820126 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.820127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.820225 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.820228 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.820295 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.820349 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":12}
I20260926 05:44:11.820480 6689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.816675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44987 (local address 127.23.213.194:33459)
0926 05:44:11.816979 (+ 304us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.816989 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:11.817048 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:11.817198 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:11.817198 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:11.817199 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:11.817261 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.817263 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.818134 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.818135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.819341 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:11.819344 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.820126 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.820127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:11.820220 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.820225 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:44:11.820295 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 05:44:11.820349 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":150,"thread_start_us":103,"threads_started":1}
I20260926 05:44:11.820817 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3968e0011bc14a20b9e912ab30ac7a50" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a89f69f706405590660e7e0f53b389" is_pre_election: true
I20260926 05:44:11.821003 6627 raft_consensus.cc:2454] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 0.
I20260926 05:44:11.821158 6357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3968e0011bc14a20b9e912ab30ac7a50" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638bbb3b54744426a64392bcb52d3387" is_pre_election: true
I20260926 05:44:11.821233 6431 leader_election.cc:302] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38a89f69f706405590660e7e0f53b389, ee15fd8ee555439cade240eddd614b42; no voters:
I20260926 05:44:11.821346 6357 raft_consensus.cc:2454] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 0.
I20260926 05:44:11.821406 6687 raft_consensus.cc:2781] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:44:11.821470 6687 raft_consensus.cc:482] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:44:11.821503 6687 raft_consensus.cc:3037] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:11.822338 6687 raft_consensus.cc:504] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.822484 6687 leader_election.cc:288] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 election: Requested vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987)
I20260926 05:44:11.822641 6627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3968e0011bc14a20b9e912ab30ac7a50" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38a89f69f706405590660e7e0f53b389"
I20260926 05:44:11.822696 6357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3968e0011bc14a20b9e912ab30ac7a50" candidate_uuid: "ee15fd8ee555439cade240eddd614b42" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "638bbb3b54744426a64392bcb52d3387"
I20260926 05:44:11.822778 6357 raft_consensus.cc:3037] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:11.822778 6627 raft_consensus.cc:3037] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:11.823690 6357 raft_consensus.cc:2454] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 1.
I20260926 05:44:11.823690 6627 raft_consensus.cc:2454] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee15fd8ee555439cade240eddd614b42 in term 1.
I20260926 05:44:11.823849 6430 leader_election.cc:302] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 638bbb3b54744426a64392bcb52d3387, ee15fd8ee555439cade240eddd614b42; no voters:
I20260926 05:44:11.823975 6687 raft_consensus.cc:2781] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:11.824143 6687 raft_consensus.cc:686] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 1 LEADER]: Becoming Leader. State: Replica: ee15fd8ee555439cade240eddd614b42, State: Running, Role: LEADER
I20260926 05:44:11.824244 6687 consensus_queue.cc:237] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:11.825011 6215 catalog_manager.cc:5495] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee15fd8ee555439cade240eddd614b42 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "ee15fd8ee555439cade240eddd614b42" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } health_report { overall_health: UNKNOWN } } }
W20260926 05:44:11.856261 6404 tablet.cc:2369] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:44:11.864657 6681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.862564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50574 (local address 127.23.213.194:37827)
0926 05:44:11.862620 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:11.862622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:11.862631 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:44:11.862666 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:11.862667 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:11.862686 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:11.862747 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:44:11.862846 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.863333 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.864001 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:11.864183 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:11.864238 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:11.864363 (+ 125us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:44:11.864469 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:44:11.864532 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:11.864563 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 05:44:11.864580 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:44:11.866519 6627 raft_consensus.cc:1261] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Refusing update from remote peer ee15fd8ee555439cade240eddd614b42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:11.866519 6357 raft_consensus.cc:1261] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Refusing update from remote peer ee15fd8ee555439cade240eddd614b42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:11.866860 6690 consensus_queue.cc:1035] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:11.867003 6687 consensus_queue.cc:1035] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:11.869305 6693 mvcc.cc:205] Tried to move back new op lower bound from 7333484346842230784 to 7333484346672640000. Current Snapshot: MvccSnapshot[applied={T|T < 7333484346842230784}]
I20260926 05:44:11.869434 6695 mvcc.cc:205] Tried to move back new op lower bound from 7333484346842230784 to 7333484346672640000. Current Snapshot: MvccSnapshot[applied={T|T < 7333484346842230784}]
I20260926 05:44:11.871572 6215 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:38466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:11.871670 6215 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:38466:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:11.872413 6215 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Sending DeleteTablet for 3 replicas of tablet 3968e0011bc14a20b9e912ab30ac7a50
I20260926 05:44:11.872766 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 6253
I20260926 05:44:11.872789 6475 tablet_service.cc:1514] Processing DeleteTablet for tablet 3968e0011bc14a20b9e912ab30ac7a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:11 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:50570
I20260926 05:44:11.872789 6607 tablet_service.cc:1514] Processing DeleteTablet for tablet 3968e0011bc14a20b9e912ab30ac7a50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:11 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42896
I20260926 05:44:11.873374 6701 tablet_replica.cc:330] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: stopping tablet replica
I20260926 05:44:11.873490 6701 raft_consensus.cc:2229] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:11.873594 6701 raft_consensus.cc:2258] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:11.874087 6701 ts_tablet_manager.cc:1905] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:11.876085 6702 tablet_replica.cc:330] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: stopping tablet replica
I20260926 05:44:11.876112 6701 ts_tablet_manager.cc:1918] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:11.876195 6702 raft_consensus.cc:2229] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:44:11.876304 6701 log.cc:1196] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/3968e0011bc14a20b9e912ab30ac7a50
I20260926 05:44:11.876310 6702 raft_consensus.cc:2258] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:11.876736 6701 ts_tablet_manager.cc:1939] T 3968e0011bc14a20b9e912ab30ac7a50 P 38a89f69f706405590660e7e0f53b389: Deleting consensus metadata
I20260926 05:44:11.876739 6702 ts_tablet_manager.cc:1905] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:11.877249 6202 catalog_manager.cc:4841] TS 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139): tablet 3968e0011bc14a20b9e912ab30ac7a50 (table test-table [id=a73a3014dd5e4528a36aaacfd9876616]) successfully deleted
I20260926 05:44:11.878715 6702 ts_tablet_manager.cc:1918] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:11.878793 6702 log.cc:1196] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/3968e0011bc14a20b9e912ab30ac7a50
I20260926 05:44:11.879042 6702 ts_tablet_manager.cc:1939] T 3968e0011bc14a20b9e912ab30ac7a50 P ee15fd8ee555439cade240eddd614b42: Deleting consensus metadata
I20260926 05:44:11.879448 6201 catalog_manager.cc:4841] TS ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827): tablet 3968e0011bc14a20b9e912ab30ac7a50 (table test-table [id=a73a3014dd5e4528a36aaacfd9876616]) successfully deleted
W20260926 05:44:11.880270 6201 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:44987 (error 108)
I20260926 05:44:11.880738 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 6408
I20260926 05:44:11.880817 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:11.880667 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:44987 (local address 127.0.0.1:36774)
0926 05:44:11.880730 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:11.880780 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:44987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 05:44:11.881872 6201 catalog_manager.cc:4584] TS 638bbb3b54744426a64392bcb52d3387 (127.23.213.193:44987): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3968e0011bc14a20b9e912ab30ac7a50: Network error: Client connection negotiation failed: client connection to 127.23.213.193:44987: connect: Connection refused (error 111)
I20260926 05:44:11.885579 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 6544
I20260926 05:44:11.890942 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 6182
I20260926 05:44:11.896114 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41399
--webserver_interface=127.23.213.254
--webserver_port=43717
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:41399
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:11.982404 6704 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:11.982626 6704 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:11.982651 6704 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:11.983894 6704 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:11.983938 6704 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:11.983955 6704 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:11.983968 6704 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:11.983980 6704 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:11.985502 6704 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:41399
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=43717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6704
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:11.985742 6704 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:11.985981 6704 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:11.990972 6710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:11.990981 6712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:11.990991 6704 server_base.cc:1034] running on GCE node
W20260926 05:44:11.990973 6709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:11.991525 6704 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:11.991837 6704 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:11.992986 6704 hybrid_clock.cc:648] HybridClock initialized: now 1790401451992972 us; error 29 us; skew 500 ppm
Sep 26 05:44:11 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401451, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:11.995571 6704 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:11.996064 6704 webserver.cc:460] Webserver started at http://127.23.213.254:43717/ using document root <none> and password file <none>
I20260926 05:44:11.996299 6704 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:11.996367 6704 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:11.997713 6704 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:11.998462 6719 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:11.998657 6704 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:11.998759 6704 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "9943514d9c374a8a83b4d597df63fbcd"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "d072bcf777ec714e00f60712af830c89"
server_key_iv: "4b66535e4ddbdaa26a85eb033100b681"
server_key_version: "encryptionkey@0"
I20260926 05:44:11.999155 6704 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:12.016620 6704 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:12.018602 6704 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:12.018774 6704 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:12.023042 6704 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:41399
I20260926 05:44:12.023110 6771 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:41399 every 8 connection(s)
I20260926 05:44:12.023523 6704 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:12.025106 6772 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:44:12.025979 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Bootstrap starting.
I20260926 05:44:12.031591 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6704
I20260926 05:44:12.031976 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:44987
--local_ip_for_outbound_sockets=127.23.213.193
--tserver_master_addrs=127.23.213.254:41399
--webserver_port=46291
--webserver_interface=127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:12.035022 6772 log.cc:824] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:12.037693 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:44:12.038079 6772 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd: Bootstrap complete.
I20260926 05:44:12.040439 6772 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:12.040676 6772 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9943514d9c374a8a83b4d597df63fbcd, State: Initialized, Role: FOLLOWER
I20260926 05:44:12.040848 6772 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:12.040925 6772 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:44:12.040987 6772 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:44:12.041042 6772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:44:12.042243 6772 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:12.042333 6772 leader_election.cc:302] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9943514d9c374a8a83b4d597df63fbcd; no voters:
I20260926 05:44:12.042580 6772 leader_election.cc:288] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 05:44:12.042618 6776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:44:12.042721 6776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [term 3 LEADER]: Becoming Leader. State: Replica: 9943514d9c374a8a83b4d597df63fbcd, State: Running, Role: LEADER
I20260926 05:44:12.042820 6776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } }
I20260926 05:44:12.042862 6772 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:12.043314 6776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9943514d9c374a8a83b4d597df63fbcd. Latest consensus state: current_term: 3 leader_uuid: "9943514d9c374a8a83b4d597df63fbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } } }
I20260926 05:44:12.043390 6776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:12.043531 6777 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9943514d9c374a8a83b4d597df63fbcd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9943514d9c374a8a83b4d597df63fbcd" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41399 } } }
I20260926 05:44:12.043602 6777 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9943514d9c374a8a83b4d597df63fbcd [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:12.043823 6780 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:12.044633 6780 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:12.044732 6780 catalog_manager.cc:1259] Loaded cluster ID: 4b055c6fd1984ceab84ae4f50bd7b864
I20260926 05:44:12.044767 6780 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 05:44:12.045243 6780 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fae22ee09ad google::LogMessage::Fail() at ??:0
@ 0x7fae22ee388a google::LogMessage::SendToLog() at ??:0
@ 0x7fae22ee04de google::LogMessage::Flush() at ??:0
@ 0x7fae22ee1a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fae2ae010f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fae2ae0a8b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fae24dae0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fae24da3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fae25c076db start_thread at ??:0
@ 0x7fae219de61f clone at ??:0
W20260926 05:44:12.129976 6774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:12.130178 6774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:12.130203 6774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:12.131395 6774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:12.131441 6774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:12.131469 6774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:44:12.132941 6774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:44987
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=46291
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:12.133205 6774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:12.133468 6774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:12.138590 6809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:12.138598 6808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:12.138598 6811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:12.139060 6774 server_base.cc:1034] running on GCE node
I20260926 05:44:12.139283 6774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:12.139529 6774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:12.140765 6774 hybrid_clock.cc:648] HybridClock initialized: now 1790401452140750 us; error 29 us; skew 500 ppm
Sep 26 05:44:12 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:12.143842 6774 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:44:12.144320 6774 webserver.cc:460] Webserver started at http://127.23.213.193:46291/ using document root <none> and password file <none>
I20260926 05:44:12.144577 6774 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:12.144651 6774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:12.146092 6774 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:12.146862 6818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:12.147063 6774 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:12.147147 6774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "638bbb3b54744426a64392bcb52d3387"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "b0785cd38a41711de3b224c8245d4790"
server_key_iv: "6f948e9c5a0581225a6092cb07f5d207"
server_key_version: "encryptionkey@0"
I20260926 05:44:12.147495 6774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:12.171784 6774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:12.174419 6774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:12.174604 6774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:12.174866 6774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:12.175343 6825 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:44:12.177479 6774 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:44:12.177558 6774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:44:12.177601 6774 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:44:12.178344 6774 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:44:12.178408 6774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:12.178501 6825 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Bootstrap starting.
I20260926 05:44:12.185703 6774 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:44987
I20260926 05:44:12.185809 6932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:44987 every 8 connection(s)
I20260926 05:44:12.186152 6774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:44:12.187795 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6774
I20260926 05:44:12.188091 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:37827
--local_ip_for_outbound_sockets=127.23.213.194
--tserver_master_addrs=127.23.213.254:41399
--webserver_port=42703
--webserver_interface=127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:12.190683 6825 log.cc:824] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:12.193815 6825 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:44:12.194188 6825 tablet_bootstrap.cc:492] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Bootstrap complete.
I20260926 05:44:12.195566 6825 ts_tablet_manager.cc:1397] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.013s
I20260926 05:44:12.197185 6825 raft_consensus.cc:348] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:12.197563 6825 raft_consensus.cc:729] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 638bbb3b54744426a64392bcb52d3387, State: Initialized, Role: FOLLOWER
I20260926 05:44:12.197732 6825 consensus_queue.cc:260] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:12.198078 6825 ts_tablet_manager.cc:1428] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
W20260926 05:44:12.275501 6935 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:44:12.186827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:40095)
0926 05:44:12.187003 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.187047 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:12.187504 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:12.275235 (+ 87731us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":103,"threads_started":1}
W20260926 05:44:12.275725 6933 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:41399 (0 consecutive failures): Network error: Failed to ping master at 127.23.213.254:41399: Client connection negotiation failed: client connection to 127.23.213.254:41399: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:44:12.276067 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.275948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:55847)
0926 05:44:12.276004 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.276049 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:44:12.276376 6935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.276282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:49229)
0926 05:44:12.276328 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.276352 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:44:12.276490 6933 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:44:12.276365 6936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:12.276695 6936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:12.276747 6936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:12.278007 6936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:12.278069 6936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:12.278115 6936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:44:12.279597 6936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:37827
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=42703
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:12.279870 6936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:12.280109 6936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:12.285239 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:12.285239 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:12.285460 6936 server_base.cc:1034] running on GCE node
W20260926 05:44:12.285307 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:12.285712 6936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:12.286031 6936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:12.287179 6936 hybrid_clock.cc:648] HybridClock initialized: now 1790401452287166 us; error 30 us; skew 500 ppm
Sep 26 05:44:12 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:12.289688 6936 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:44:12.290217 6936 webserver.cc:460] Webserver started at http://127.23.213.194:42703/ using document root <none> and password file <none>
I20260926 05:44:12.290465 6936 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:12.290539 6936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:12.291975 6936 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:44:12.292680 6953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:12.292907 6936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:12.293020 6936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "ee15fd8ee555439cade240eddd614b42"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "4511c0ca2cc16629e9c66ae1e249a4cb"
server_key_iv: "6358f2b48c4fd41113fe50ad62e0063f"
server_key_version: "encryptionkey@0"
I20260926 05:44:12.293371 6936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:12.341368 6936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:12.343647 6936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:12.343823 6936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:12.344080 6936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:12.344404 6936 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:12.344462 6936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:12.344498 6936 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:12.344527 6936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:12.351001 6936 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:37827
I20260926 05:44:12.351044 7066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:37827 every 8 connection(s)
I20260926 05:44:12.351454 6936 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:44:12.352545 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.352104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:33731)
0926 05:44:12.352260 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.352395 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":68,"threads_started":1}
W20260926 05:44:12.353018 7067 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:41399 (0 consecutive failures): Network error: Failed to ping master at 127.23.213.254:41399: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
I20260926 05:44:12.353295 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.353208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:59497)
0926 05:44:12.353253 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.353279 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 05:44:12.353631 7069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.353555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:34027)
0926 05:44:12.353596 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.353617 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:44:12.353772 7067 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:44:12.354050 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 6936
I20260926 05:44:12.354341 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:45139
--local_ip_for_outbound_sockets=127.23.213.195
--tserver_master_addrs=127.23.213.254:41399
--webserver_port=34199
--webserver_interface=127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:45165
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:12.435489 7070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:12.435751 7070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:12.435789 7070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:12.436976 7070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:12.437065 7070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:12.437114 7070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:44:12.438657 7070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45165
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:45139
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=34199
--tserver_master_addrs=127.23.213.254:41399
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7070
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:12.438951 7070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:12.439210 7070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:12.444566 7078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:12.444579 7075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:12.444690 7076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:12.444885 7070 server_base.cc:1034] running on GCE node
I20260926 05:44:12.445102 7070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:12.445322 7070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:12.446496 7070 hybrid_clock.cc:648] HybridClock initialized: now 1790401452446488 us; error 52 us; skew 500 ppm
Sep 26 05:44:12 dist-test-slave-1fdt krb5kdc[5636](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:12.449185 7070 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:44:12.449698 7070 webserver.cc:460] Webserver started at http://127.23.213.195:34199/ using document root <none> and password file <none>
I20260926 05:44:12.449980 7070 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:12.450030 7070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:12.451423 7070 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:12.452032 7085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:12.452216 7070 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:12.452324 7070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "38a89f69f706405590660e7e0f53b389"
format_stamp: "Formatted at 2026-09-26 05:44:10 on dist-test-slave-1fdt"
server_key: "419d39b96fd577e6fb642e28f5ebe858"
server_key_iv: "0f6067ce899fb04d784f5db34da9d0e5"
server_key_version: "encryptionkey@0"
I20260926 05:44:12.452677 7070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:12.476686 7070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:12.478816 7070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:12.478996 7070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:12.479264 7070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:12.479590 7070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:12.479648 7070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:12.479691 7070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:12.479722 7070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:12.485796 7070 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:45139
I20260926 05:44:12.485841 7198 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:45139 every 8 connection(s)
I20260926 05:44:12.486207 7070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:44:12.487272 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.486877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:56369)
0926 05:44:12.487041 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.487128 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
W20260926 05:44:12.487681 7199 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:41399 (0 consecutive failures): Network error: Failed to ping master at 127.23.213.254:41399: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
I20260926 05:44:12.487972 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.487870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:45229)
0926 05:44:12.487924 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.487952 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 05:44:12.488272 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:12.488190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:60449)
0926 05:44:12.488234 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:12.488257 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:44:12.488394 7199 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:44:12.489333 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 7070
I20260926 05:44:13.277132 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:13.276857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:48109)
0926 05:44:13.277030 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:13.277072 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":62,"threads_started":1}
I20260926 05:44:13.354346 7204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:13.354136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:50769)
0926 05:44:13.354257 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:13.354295 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":54,"threads_started":1}
I20260926 05:44:13.488970 7205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:13.488750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:37765)
0926 05:44:13.488877 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:13.488918 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260926 05:44:13.575917 7206 raft_consensus.cc:482] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:13.576033 7206 raft_consensus.cc:504] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:13.576400 7206 leader_election.cc:288] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827)
Sep 26 05:44:13 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.194@KRBTEST.COM
Sep 26 05:44:13 dist-test-slave-1fdt krb5kdc[5636](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401452, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.195@KRBTEST.COM
I20260926 05:44:13.591440 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:13.576608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:37827 (local address 127.23.213.193:35055)
0926 05:44:13.576678 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:13.576691 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:13.576777 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:13.577657 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:13.577660 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:13.578087 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:13.578278 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:13.578283 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.579370 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.579371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:13.580233 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:13.580235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.580932 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.580932 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:13.580983 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:13.585948 (+ 4965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:13.585953 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:13.587571 (+ 1618us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:13.589686 (+ 2115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:13.589693 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:13.589696 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:13.589970 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:13.590257 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:13.590258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:13.590259 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:13.590419 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:13.591032 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:13.591039 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:13.591167 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 05:44:13.591243 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:44:13.591365 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48,"mutex_wait_us":2}
I20260926 05:44:13.591666 7208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:13.576749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:35055 (local address 127.23.213.194:37827)
0926 05:44:13.576926 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:13.576939 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:13.576981 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:44:13.577403 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:13.577405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:13.577442 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:13.577624 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:13.578403 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.579252 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.580358 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.580768 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.580894 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:13.587710 (+ 6816us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:13.587715 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:13.587715 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:13.587726 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:13.589574 (+ 1848us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:13.590056 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:13.590060 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:13.590060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:13.590181 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:13.590507 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:13.590508 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:13.590508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:13.590956 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:13.591041 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:13.591351 (+ 310us) server_negotiation.cc:299] Negotiation successful
0926 05:44:13.591490 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":84,"threads_started":1}
I20260926 05:44:13.591842 7207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:13.576624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45139 (local address 127.23.213.193:41769)
0926 05:44:13.576718 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:13.576729 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:13.576796 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:13.577658 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:13.577661 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:13.578088 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:13.578278 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:13.578284 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.579337 (+ 1053us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.579339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:13.580235 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:13.580238 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.580862 (+ 624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.580863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:13.580982 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:13.585910 (+ 4928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:13.585919 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:13.588043 (+ 2124us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:13.590086 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:13.590086 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:13.590087 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:13.590282 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:13.590545 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:13.590545 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:13.590546 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:13.590606 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:13.591554 (+ 948us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:13.591555 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:13.591597 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 05:44:13.591666 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 05:44:13.591760 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":71,"thread_start_us":45,"threads_started":1}
I20260926 05:44:13.592104 7209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:13.576830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:41769 (local address 127.23.213.195:45139)
0926 05:44:13.576946 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:13.576954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:13.576981 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:44:13.577398 (+ 417us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:13.577401 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:13.577442 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:13.577636 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:13.578406 (+ 770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.579232 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.580337 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:13.580769 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:13.580851 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:13.588184 (+ 7333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:13.588188 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:13.588189 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:13.588202 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:13.590029 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:13.590369 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:13.590371 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:13.590372 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:13.590490 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:13.590683 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:13.590684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:13.590685 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:13.590956 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:13.591050 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:13.591745 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 05:44:13.591922 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":47,"threads_started":1}
W20260926 05:44:13.592376 6821 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
W20260926 05:44:13.592690 6822 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
I20260926 05:44:13.592730 6822 leader_election.cc:302] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 638bbb3b54744426a64392bcb52d3387; no voters: 38a89f69f706405590660e7e0f53b389, ee15fd8ee555439cade240eddd614b42
I20260926 05:44:13.592834 7206 raft_consensus.cc:2726] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:44:14.278121 7210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:14.277889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:59595)
0926 05:44:14.278030 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:14.278074 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":61,"threads_started":1}
I20260926 05:44:14.355091 7211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:14.354869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:39581)
0926 05:44:14.355008 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:14.355047 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260926 05:44:14.489708 7212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:14.489467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:54915)
0926 05:44:14.489624 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:14.489666 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":73,"threads_started":1}
I20260926 05:44:15.279121 7213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:15.278815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:49037)
0926 05:44:15.279013 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:15.279068 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":73,"threads_started":1}
I20260926 05:44:15.355865 7214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:15.355654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:38135)
0926 05:44:15.355771 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:15.355808 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260926 05:44:15.490456 7215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:15.490231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:45199)
0926 05:44:15.490362 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:15.490404 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260926 05:44:15.756799 7216 raft_consensus.cc:482] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:15.756924 7216 raft_consensus.cc:504] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:15.757133 7216 leader_election.cc:288] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827)
W20260926 05:44:15.757582 6821 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
W20260926 05:44:15.757652 6822 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
I20260926 05:44:15.757781 6821 leader_election.cc:302] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 638bbb3b54744426a64392bcb52d3387; no voters: 38a89f69f706405590660e7e0f53b389, ee15fd8ee555439cade240eddd614b42
I20260926 05:44:15.757879 7216 raft_consensus.cc:2726] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:44:16.280010 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:16.279711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:38373)
0926 05:44:16.279904 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:16.279952 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":60,"threads_started":1}
I20260926 05:44:16.356577 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:16.356369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:55053)
0926 05:44:16.356491 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:16.356534 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260926 05:44:16.491196 7219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:16.490970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:56643)
0926 05:44:16.491115 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:16.491154 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":61,"threads_started":1}
I20260926 05:44:17.280979 7220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:17.280639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:53323)
0926 05:44:17.280886 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:17.280931 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":227,"thread_start_us":80,"threads_started":1}
I20260926 05:44:17.357235 7221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:17.357036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:56999)
0926 05:44:17.357156 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:17.357189 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":57,"threads_started":1}
I20260926 05:44:17.492003 7222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:17.491713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:50849)
0926 05:44:17.491910 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:17.491952 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":140,"threads_started":1}
I20260926 05:44:17.770684 7223 raft_consensus.cc:482] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:17.770778 7223 raft_consensus.cc:504] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:17.770970 7223 leader_election.cc:288] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827)
W20260926 05:44:17.771404 6821 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
W20260926 05:44:17.771493 6822 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
I20260926 05:44:17.771588 6822 leader_election.cc:302] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 638bbb3b54744426a64392bcb52d3387; no voters: 38a89f69f706405590660e7e0f53b389, ee15fd8ee555439cade240eddd614b42
I20260926 05:44:17.771687 7223 raft_consensus.cc:2726] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:44:18.281816 7224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:18.281524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:57901)
0926 05:44:18.281668 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:18.281711 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260926 05:44:18.357951 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:18.357715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:52087)
0926 05:44:18.357871 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:18.357908 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":100,"threads_started":1}
I20260926 05:44:18.492875 7226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:18.492619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:45131)
0926 05:44:18.492756 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:18.492799 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":93,"threads_started":1}
I20260926 05:44:19.282603 7227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:19.282362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:49581)
0926 05:44:19.282497 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:19.282557 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260926 05:44:19.358688 7228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:19.358457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:38243)
0926 05:44:19.358602 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:19.358644 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":73,"threads_started":1}
I20260926 05:44:19.493549 7229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:19.493331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:48065)
0926 05:44:19.493464 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:19.493503 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":59,"threads_started":1}
I20260926 05:44:20.283403 7230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:20.283090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:55273)
0926 05:44:20.283271 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:20.283318 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":66,"threads_started":1}
I20260926 05:44:20.359508 7231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:20.359236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:34045)
0926 05:44:20.359412 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:20.359456 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":71,"threads_started":1}
I20260926 05:44:20.494305 7232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:20.494068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:36351)
0926 05:44:20.494209 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:20.494257 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":61,"threads_started":1}
I20260926 05:44:21.284116 7233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:21.283917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:56163)
0926 05:44:21.284038 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:21.284071 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":63,"threads_started":1}
I20260926 05:44:21.360209 7234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:21.360003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:58261)
0926 05:44:21.360124 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:21.360165 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260926 05:44:21.494987 7235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:21.494760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:47849)
0926 05:44:21.494893 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:21.494933 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260926 05:44:22.284998 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:22.284614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:48151)
0926 05:44:22.284870 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:22.284941 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":74,"threads_started":1}
I20260926 05:44:22.360966 7237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:22.360741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:45911)
0926 05:44:22.360872 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:22.360916 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":58,"threads_started":1}
I20260926 05:44:22.495708 7238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:22.495478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:37467)
0926 05:44:22.495627 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:22.495665 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":70,"threads_started":1}
I20260926 05:44:22.630481 7239 raft_consensus.cc:482] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:22.630609 7239 raft_consensus.cc:504] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "38a89f69f706405590660e7e0f53b389" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45139 } } peers { permanent_uuid: "ee15fd8ee555439cade240eddd614b42" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 37827 } } peers { permanent_uuid: "638bbb3b54744426a64392bcb52d3387" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 44987 } }
I20260926 05:44:22.630831 7239 leader_election.cc:288] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139), ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827)
W20260926 05:44:22.631269 6821 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer ee15fd8ee555439cade240eddd614b42 (127.23.213.194:37827): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
W20260926 05:44:22.631414 6822 leader_election.cc:341] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 38a89f69f706405590660e7e0f53b389 (127.23.213.195:45139): Not found: Tablet not found: 3968e0011bc14a20b9e912ab30ac7a50
I20260926 05:44:22.631464 6822 leader_election.cc:302] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 638bbb3b54744426a64392bcb52d3387; no voters: 38a89f69f706405590660e7e0f53b389, ee15fd8ee555439cade240eddd614b42
I20260926 05:44:22.631572 7239 raft_consensus.cc:2726] T 3968e0011bc14a20b9e912ab30ac7a50 P 638bbb3b54744426a64392bcb52d3387 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:44:23.285959 7240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:23.285692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:40011)
0926 05:44:23.285881 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:23.285914 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":173,"thread_start_us":130,"threads_started":1}
I20260926 05:44:23.361683 7241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:23.361440 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:45241)
0926 05:44:23.361588 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:23.361632 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260926 05:44:23.496488 7242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:23.496258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:53837)
0926 05:44:23.496399 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:23.496439 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":66,"threads_started":1}
I20260926 05:44:24.286639 7243 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:24.286397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:55833)
0926 05:44:24.286551 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:24.286593 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":73,"threads_started":1}
I20260926 05:44:24.362447 7244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:24.362250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:46441)
0926 05:44:24.362372 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:24.362403 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260926 05:44:24.497248 7245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:24.497023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:48189)
0926 05:44:24.497148 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:24.497192 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
I20260926 05:44:25.287421 7246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:25.287167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:54595)
0926 05:44:25.287333 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:25.287375 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":77,"threads_started":1}
I20260926 05:44:25.363204 7247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:25.362943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:49051)
0926 05:44:25.363112 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:25.363156 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":78,"threads_started":1}
I20260926 05:44:25.498013 7248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:25.497732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:48425)
0926 05:44:25.497920 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:25.497964 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":67,"threads_started":1}
I20260926 05:44:26.288147 7249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:26.287921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:40895)
0926 05:44:26.288057 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:26.288099 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260926 05:44:26.364038 7250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:26.363807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:36823)
0926 05:44:26.363947 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:26.363991 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260926 05:44:26.498739 7251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:26.498517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:46491)
0926 05:44:26.498652 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:26.498690 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":69,"threads_started":1}
I20260926 05:44:27.288992 7252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:27.288748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.193:43987)
0926 05:44:27.288896 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:27.288933 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260926 05:44:27.364800 7253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:27.364568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.194:38339)
0926 05:44:27.364719 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:27.364756 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":74,"threads_started":1}
I20260926 05:44:27.497998 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 6774
I20260926 05:44:27.499500 7254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:27.499277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41399 (local address 127.23.213.195:35045)
0926 05:44:27.499405 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:27.499442 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260926 05:44:27.502830 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 6936
I20260926 05:44:27.506928 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 7070
2026-09-26T05:44:27Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18310 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:44:27 dist-test-slave-1fdt krb5kdc[7257](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:44:27 dist-test-slave-1fdt krb5kdc[7257](info): set up 2 sockets
Sep 26 05:44:27 dist-test-slave-1fdt krb5kdc[7257](info): commencing operation
krb5kdc: starting...
W20260926 05:44:28.545976 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:44:28 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:44:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:44:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:44:28.667135 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43631
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:28.759490 7270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:28.759696 7270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:28.759717 7270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:28.760859 7270 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:44:28.760901 7270 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:44:28.760914 7270 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:44:28.760926 7270 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:28.760938 7270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:28.760951 7270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:28.761001 7270 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:28.761019 7270 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:28.762534 7270 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:43631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:28.762787 7270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:28.763023 7270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:28.767835 7275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:28.767889 7278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:28.767835 7276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:28.768198 7270 server_base.cc:1034] running on GCE node
I20260926 05:44:28.768462 7270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:28.768760 7270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:28.769901 7270 hybrid_clock.cc:648] HybridClock initialized: now 1790401468769876 us; error 35 us; skew 500 ppm
Sep 26 05:44:28 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:28.772377 7270 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:28.772833 7270 webserver.cc:460] Webserver started at http://127.23.213.254:42997/ using document root <none> and password file <none>
I20260926 05:44:28.773064 7270 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:28.773133 7270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:28.773289 7270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:28.774441 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "38aa5f402ab4424981662b7716c49a29"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "2c0f75ee734ee0c93890c78a10248625"
server_key_iv: "f5e23dad9381859e9825d325925f9b6e"
server_key_version: "encryptionkey@0"
I20260926 05:44:28.774816 7270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "38aa5f402ab4424981662b7716c49a29"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "2c0f75ee734ee0c93890c78a10248625"
server_key_iv: "f5e23dad9381859e9825d325925f9b6e"
server_key_version: "encryptionkey@0"
I20260926 05:44:28.776157 7270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:44:28.776865 7285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:28.777047 7270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:28.777153 7270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "38aa5f402ab4424981662b7716c49a29"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "2c0f75ee734ee0c93890c78a10248625"
server_key_iv: "f5e23dad9381859e9825d325925f9b6e"
server_key_version: "encryptionkey@0"
I20260926 05:44:28.777235 7270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:28.792918 7270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:28.794945 7270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:28.795120 7270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:28.799211 7270 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:43631
I20260926 05:44:28.799249 7337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:43631 every 8 connection(s)
I20260926 05:44:28.799636 7270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:28.800536 7338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:28.802845 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Bootstrap starting.
I20260926 05:44:28.803402 7338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:28.803592 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 7270
I20260926 05:44:28.803704 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:44:28.803756 7338 log.cc:824] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:28.803923 24407 external_mini_cluster.cc:1428] Setting key 06255fc45964cae312baeda03a0eac0f
I20260926 05:44:28.804679 7338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: No bootstrap required, opened a new log
I20260926 05:44:28.807413 7338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:28.807590 7338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:28.807624 7338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38aa5f402ab4424981662b7716c49a29, State: Initialized, Role: FOLLOWER
I20260926 05:44:28.807753 7338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:28.807821 7338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:44:28.807848 7338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:44:28.807890 7338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:28.808511 7338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:28.808631 7338 leader_election.cc:302] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38aa5f402ab4424981662b7716c49a29; no voters:
I20260926 05:44:28.808856 7338 leader_election.cc:288] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:44:28.808887 7343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:28.809024 7343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 LEADER]: Becoming Leader. State: Replica: 38aa5f402ab4424981662b7716c49a29, State: Running, Role: LEADER
I20260926 05:44:28.809175 7338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:28.809166 7343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:28.809661 7345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38aa5f402ab4424981662b7716c49a29. Latest consensus state: current_term: 1 leader_uuid: "38aa5f402ab4424981662b7716c49a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } } }
I20260926 05:44:28.809764 7345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: This master's current role is: LEADER
Sep 26 05:44:28 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401468, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:28.810776 7349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:28.811290 7344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "38aa5f402ab4424981662b7716c49a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } } }
I20260926 05:44:28.811467 7344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:28.811506 7349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:28.813458 7349 catalog_manager.cc:1347] Generated new cluster ID: d7d27a95c5b74035bac17b5bf7383021
I20260926 05:44:28.813510 7349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:44:28.815466 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:28.804519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42564 (local address 127.23.213.254:43631)
0926 05:44:28.804713 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:28.804720 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:28.804763 (+ 43us) server_negotiation.cc:407] Connection header received
0926 05:44:28.805195 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:28.805201 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:28.805248 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:28.805361 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:28.806194 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:28.807077 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:28.808143 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:28.808544 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:28.808666 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:28.811587 (+ 2921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:28.811591 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:28.811591 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:28.811603 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:28.813568 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:28.813950 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:28.813953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:28.813953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:28.814054 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:28.814230 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:28.814231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:28.814249 (+ 18us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:28.814534 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:28.814661 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:28.815116 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 05:44:28.815243 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":102,"threads_started":1}
I20260926 05:44:28.820070 7349 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:44:28.820667 7349 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:28.838316 7349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Generated new TSK 0
I20260926 05:44:28.838577 7349 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:44:28.865640 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:43631
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:28.946476 7364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:28.946758 7364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:28.946784 7364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:28.947896 7364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:28.947929 7364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:28.947954 7364 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:44:28.949302 7364 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:28.949476 7364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:28.949644 7364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:28.954583 7370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:28.954574 7372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:28.954581 7369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:28.954959 7364 server_base.cc:1034] running on GCE node
I20260926 05:44:28.955176 7364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:28.955427 7364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:28.956619 7364 hybrid_clock.cc:648] HybridClock initialized: now 1790401468956602 us; error 23 us; skew 500 ppm
Sep 26 05:44:28 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:28.959252 7364 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:44:28.959775 7364 webserver.cc:460] Webserver started at http://127.23.213.193:42775/ using document root <none> and password file <none>
I20260926 05:44:28.960026 7364 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:28.960103 7364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:28.960271 7364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:28.961378 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "659784a0e78748f7a1ce628fbd3f5440"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "d79192cb49ecd28beefc5182d8b75457"
server_key_iv: "5c456bb8fc924d68106e6cc9566bfd18"
server_key_version: "encryptionkey@0"
I20260926 05:44:28.961933 7364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "659784a0e78748f7a1ce628fbd3f5440"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "d79192cb49ecd28beefc5182d8b75457"
server_key_iv: "5c456bb8fc924d68106e6cc9566bfd18"
server_key_version: "encryptionkey@0"
I20260926 05:44:28.963415 7364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:44:28.964131 7379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:28.964313 7364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:28.964395 7364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "659784a0e78748f7a1ce628fbd3f5440"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "d79192cb49ecd28beefc5182d8b75457"
server_key_iv: "5c456bb8fc924d68106e6cc9566bfd18"
server_key_version: "encryptionkey@0"
I20260926 05:44:28.964442 7364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:28.982963 7364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:28.985136 7364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:28.985246 7364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:28.985431 7364 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:28.985785 7364 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:28.985829 7364 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:28.985868 7364 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:28.985893 7364 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:28.991595 7364 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:39559
I20260926 05:44:28.991698 7492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:39559 every 8 connection(s)
I20260926 05:44:28.991981 7364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
Sep 26 05:44:28 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401468, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:29.000623 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:28.992687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:56809 (local address 127.23.213.254:43631)
0926 05:44:28.992735 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:28.992738 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:28.992963 (+ 225us) server_negotiation.cc:407] Connection header received
0926 05:44:28.993440 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:28.993441 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:28.993451 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:28.993500 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:28.994483 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:28.994992 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:28.995937 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:28.996140 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:28.996190 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:28.998271 (+ 2081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:28.998273 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:28.998273 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:28.998279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:28.999409 (+ 1130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:28.999764 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:28.999765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:28.999765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:28.999813 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:29.000028 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:29.000029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:29.000029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:29.000131 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:29.000180 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.000406 (+ 226us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.000478 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:44:29.000666 7495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:28.992682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:56809)
0926 05:44:28.992843 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:28.992878 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:28.993356 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:28.993558 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:28.993561 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:28.994213 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:28.994402 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:28.994408 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:28.995069 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:28.995070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:28.995860 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:28.995861 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:28.996209 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:28.996209 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:28.996309 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:28.996738 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:28.996746 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:28.998180 (+ 1434us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:28.999477 (+ 1297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:28.999483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:28.999485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:28.999700 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:28.999868 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:28.999869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:28.999869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:28.999974 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:29.000183 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:29.000187 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:29.000301 (+ 114us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.000382 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.000500 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":89,"threads_started":1}
I20260926 05:44:29.000705 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 7364
I20260926 05:44:29.000811 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:44:29.001008 24407 external_mini_cluster.cc:1428] Setting key fdbbb8e163c6f8a1c4d67ba8f29d7e7d
I20260926 05:44:29.001338 7493 heartbeater.cc:347] Connected to a master server at 127.23.213.254:43631
I20260926 05:44:29.001458 7493 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:29.001657 7493 heartbeater.cc:511] Master 127.23.213.254:43631 requested a full tablet report, sending...
I20260926 05:44:29.002218 7302 ts_manager.cc:194] Registered new tserver with Master: 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193:39559)
I20260926 05:44:29.002841 7302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:56809
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:44:29.024885 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:43631
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:29.105206 7498 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:29.105427 7498 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:29.105449 7498 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:29.106616 7498 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:29.106655 7498 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:29.106680 7498 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:44:29.108065 7498 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7498
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:29.108239 7498 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:29.108403 7498 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:29.113368 7503 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:29.113436 7504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:29.113412 7506 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:29.113489 7498 server_base.cc:1034] running on GCE node
I20260926 05:44:29.113914 7498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:29.114166 7498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:29.115324 7498 hybrid_clock.cc:648] HybridClock initialized: now 1790401469115313 us; error 31 us; skew 500 ppm
Sep 26 05:44:29 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:29.117816 7498 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:44:29.118245 7498 webserver.cc:460] Webserver started at http://127.23.213.194:36257/ using document root <none> and password file <none>
I20260926 05:44:29.118480 7498 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:29.118549 7498 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:29.118702 7498 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:29.119766 7498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "b1173c69126f41879a100eaefc15dae7"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "2fef76be80067e765856af9c264ac456"
server_key_iv: "6f2e6f3a16f5d74ab5e4b4a6ee3d00ea"
server_key_version: "encryptionkey@0"
I20260926 05:44:29.120138 7498 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "b1173c69126f41879a100eaefc15dae7"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "2fef76be80067e765856af9c264ac456"
server_key_iv: "6f2e6f3a16f5d74ab5e4b4a6ee3d00ea"
server_key_version: "encryptionkey@0"
I20260926 05:44:29.121435 7498 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:44:29.122262 7513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:29.122447 7498 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:29.122551 7498 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "b1173c69126f41879a100eaefc15dae7"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "2fef76be80067e765856af9c264ac456"
server_key_iv: "6f2e6f3a16f5d74ab5e4b4a6ee3d00ea"
server_key_version: "encryptionkey@0"
I20260926 05:44:29.122624 7498 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:29.137984 7498 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:29.139906 7498 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:29.140074 7498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:29.140309 7498 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:29.140623 7498 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:29.140677 7498 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:29.140712 7498 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:29.140740 7498 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:29.146605 7498 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:34749
I20260926 05:44:29.146657 7626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:34749 every 8 connection(s)
I20260926 05:44:29.147001 7498 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:44:29.149837 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 7498
I20260926 05:44:29.149950 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:44:29.150147 24407 external_mini_cluster.cc:1428] Setting key 05c55c94aa2c545c727c85b60c60ee7c
Sep 26 05:44:29 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:29.157199 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.147702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:48731 (local address 127.23.213.254:43631)
0926 05:44:29.147785 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.147787 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.147979 (+ 192us) server_negotiation.cc:407] Connection header received
0926 05:44:29.148504 (+ 525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.148505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.148514 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.148564 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:29.149471 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.150256 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.151432 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.151757 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.151832 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.154261 (+ 2429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:29.154262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:29.154263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:29.154271 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:29.155730 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:29.156185 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:29.156186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:29.156187 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:29.156248 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:29.156511 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:29.156512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:29.156512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:29.156631 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:29.156707 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.157025 (+ 318us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.157110 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 05:44:29.157343 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.147666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:48731)
0926 05:44:29.147857 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.147895 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:29.148412 (+ 517us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:29.148622 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:29.148626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:29.149185 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:29.149392 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.149397 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.150359 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.150360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.151343 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.151346 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.151854 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.151855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.151955 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.152527 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:29.152538 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:29.154160 (+ 1622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:29.155849 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:29.155856 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:29.155858 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:29.156090 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:29.156322 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:29.156323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:29.156323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:29.156435 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:29.156700 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:29.156706 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:29.156852 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.156957 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.157120 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":106,"threads_started":1}
I20260926 05:44:29.157820 7627 heartbeater.cc:347] Connected to a master server at 127.23.213.254:43631
I20260926 05:44:29.157925 7627 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:29.158106 7627 heartbeater.cc:511] Master 127.23.213.254:43631 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
I20260926 05:44:29.158643 7302 ts_manager.cc:194] Registered new tserver with Master: b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
I20260926 05:44:29.159276 7302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:48731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:44:29.179216 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:43631
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:29.260970 7632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:29.261214 7632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:29.261250 7632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:29.262428 7632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:29.262509 7632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:29.262558 7632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:44:29.264029 7632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:29.264304 7632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:29.264542 7632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:29.269805 7638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:29.269809 7640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:29.270125 7637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:29.270506 7632 server_base.cc:1034] running on GCE node
I20260926 05:44:29.270745 7632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:29.271046 7632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:29.272192 7632 hybrid_clock.cc:648] HybridClock initialized: now 1790401469272172 us; error 31 us; skew 500 ppm
Sep 26 05:44:29 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:29.274742 7632 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:44:29.275213 7632 webserver.cc:460] Webserver started at http://127.23.213.195:45509/ using document root <none> and password file <none>
I20260926 05:44:29.275451 7632 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:29.275502 7632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:29.275666 7632 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:44:29.276718 7632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "91ba731fdedb5130452f60cac627a828"
server_key_iv: "82a7d3cc4a318be0d859f07b0e0b8029"
server_key_version: "encryptionkey@0"
I20260926 05:44:29.277089 7632 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "91ba731fdedb5130452f60cac627a828"
server_key_iv: "82a7d3cc4a318be0d859f07b0e0b8029"
server_key_version: "encryptionkey@0"
I20260926 05:44:29.278586 7632 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:44:29.279434 7647 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:29.279663 7632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:44:29.279768 7632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "91ba731fdedb5130452f60cac627a828"
server_key_iv: "82a7d3cc4a318be0d859f07b0e0b8029"
server_key_version: "encryptionkey@0"
I20260926 05:44:29.279901 7632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:29.313916 7632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:29.316223 7632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:29.316385 7632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:29.316614 7632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:29.316921 7632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:29.316975 7632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:29.317028 7632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:29.317052 7632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:29.322770 7632 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:34903
I20260926 05:44:29.322814 7760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:34903 every 8 connection(s)
I20260926 05:44:29.323150 7632 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:44:29.325680 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 7632
I20260926 05:44:29.325800 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:44:29.325979 24407 external_mini_cluster.cc:1428] Setting key bb905935f4f17b1a6f054ae0ec0d8202
Sep 26 05:44:29 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401469, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:29.332976 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.323875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:59939 (local address 127.23.213.254:43631)
0926 05:44:29.323936 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.323938 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.324167 (+ 229us) server_negotiation.cc:407] Connection header received
0926 05:44:29.324627 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.324628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.324638 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.324690 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:29.325699 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.326556 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.327645 (+ 1089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.327886 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.327959 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.330119 (+ 2160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:29.330120 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:29.330120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:29.330127 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:29.331618 (+ 1491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:29.332019 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:29.332020 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:29.332020 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:29.332071 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:29.332295 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:29.332296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:29.332296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:29.332433 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:29.332490 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.332838 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.332910 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:44:29.333190 7763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.323867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:59939)
0926 05:44:29.324045 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.324082 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:29.324548 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:29.324768 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:29.324771 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:29.325396 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:29.325600 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.325605 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.326661 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.326662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.327545 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.327547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.327969 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.327970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.328071 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.328550 (+ 479us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:29.328557 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:29.330003 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:29.331712 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:29.331718 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:29.331720 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:29.331947 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:29.332132 (+ 185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:29.332132 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:29.332132 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:29.332234 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:29.332532 (+ 298us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:29.332537 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:29.332699 (+ 162us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.332800 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.332932 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":92,"threads_started":1}
I20260926 05:44:29.333771 7761 heartbeater.cc:347] Connected to a master server at 127.23.213.254:43631
I20260926 05:44:29.333871 7761 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:29.334040 7761 heartbeater.cc:511] Master 127.23.213.254:43631 requested a full tablet report, sending...
I20260926 05:44:29.334412 7302 ts_manager.cc:194] Registered new tserver with Master: 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903)
I20260926 05:44:29.334802 7302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:59939
I20260926 05:44:29.337785 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:44:29.345184 7342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.339227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42570 (local address 127.23.213.254:43631)
0926 05:44:29.339309 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.339312 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.339432 (+ 120us) server_negotiation.cc:407] Connection header received
0926 05:44:29.339520 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.339521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.339532 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.339588 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:29.340146 (+ 558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.340641 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.341357 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.341567 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.341651 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.342811 (+ 1160us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:29.342812 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:29.342812 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:29.342819 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:29.343936 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:29.344300 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:29.344301 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:29.344302 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:29.344362 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:29.344600 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:29.344601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:29.344601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:29.344752 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:29.344858 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.344999 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.345084 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 05:44:29.346719 7302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42570:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:44:29.348351 7302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:44:29.354981 7772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.350727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:34903 (local address 127.0.0.1:33560)
0926 05:44:29.351155 (+ 428us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.351167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:29.351223 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:29.351630 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:29.351631 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:29.351633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:29.351693 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.351695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.352566 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.352567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.353754 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.353777 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.354616 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.354617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.354737 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.354751 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.354841 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.354900 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":89,"threads_started":1}
I20260926 05:44:29.354995 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.350803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33560 (local address 127.23.213.195:34903)
0926 05:44:29.351388 (+ 585us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.351393 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.351403 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:44:29.351450 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.351452 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.351469 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.351546 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:29.351754 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.352463 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.353873 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.354517 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.354607 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.354699 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.354870 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.354919 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":79,"threads_started":1}
I20260926 05:44:29.355403 7773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.350726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:39559 (local address 127.0.0.1:43184)
0926 05:44:29.350887 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.350929 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:29.351005 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:29.351464 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:29.351467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:29.351469 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:29.351671 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.351673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.352990 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.352991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.354095 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.354098 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.355190 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.355191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.355252 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.355256 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.355288 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.355327 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":104,"thread_start_us":46,"threads_started":1}
I20260926 05:44:29.355753 7695 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad9c7f7d604d778cbe9b7ff0dda379 (DEFAULT_TABLE table=test-table [id=908a71f0496b4a238d9ea2ec033f0061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:29.355893 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.350995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43184 (local address 127.23.213.193:39559)
0926 05:44:29.351290 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.351294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.351302 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:44:29.351338 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.351340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.351356 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.351560 (+ 204us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:29.351740 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.352877 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.354188 (+ 1311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.354954 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.355649 (+ 695us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.355723 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.355749 (+ 26us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.355801 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":86,"threads_started":1}
I20260926 05:44:29.356140 7695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad9c7f7d604d778cbe9b7ff0dda379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:29.356508 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.350736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:34749 (local address 127.0.0.1:40396)
0926 05:44:29.351157 (+ 421us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.351167 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:29.351226 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:29.351464 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:29.351467 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:29.351471 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:29.351578 (+ 107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.351580 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.353834 (+ 2254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.353835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.354853 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.354856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.356299 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.356300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.356370 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.356375 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.356403 (+ 28us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.356444 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"mutex_wait_us":20,"thread_start_us":85,"threads_started":1}
I20260926 05:44:29.356660 7427 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad9c7f7d604d778cbe9b7ff0dda379 (DEFAULT_TABLE table=test-table [id=908a71f0496b4a238d9ea2ec033f0061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:29.356904 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.350972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40396 (local address 127.23.213.194:34749)
0926 05:44:29.351169 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.351173 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.351219 (+ 46us) server_negotiation.cc:407] Connection header received
0926 05:44:29.351296 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.351299 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.351317 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.351759 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:29.351838 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.352880 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.354955 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.356197 (+ 1242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.356689 (+ 492us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.356751 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.356779 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.356825 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":90,"threads_started":1}
I20260926 05:44:29.357024 7427 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad9c7f7d604d778cbe9b7ff0dda379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:29.357559 7561 tablet_service.cc:1467] Processing CreateTablet for tablet 00ad9c7f7d604d778cbe9b7ff0dda379 (DEFAULT_TABLE table=test-table [id=908a71f0496b4a238d9ea2ec033f0061]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:29.357952 7561 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00ad9c7f7d604d778cbe9b7ff0dda379. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:29.359772 7778 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Bootstrap starting.
I20260926 05:44:29.360124 7779 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap starting.
I20260926 05:44:29.360528 7778 tablet_bootstrap.cc:654] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:29.360777 7779 tablet_bootstrap.cc:654] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:29.361122 7780 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Bootstrap starting.
I20260926 05:44:29.361382 7779 log.cc:824] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:29.361382 7778 log.cc:824] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:29.361780 7780 tablet_bootstrap.cc:654] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:29.362314 7778 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: No bootstrap required, opened a new log
I20260926 05:44:29.362344 7780 log.cc:824] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:29.362393 7778 ts_tablet_manager.cc:1397] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:44:29.362490 7779 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: No bootstrap required, opened a new log
I20260926 05:44:29.362588 7779 ts_tablet_manager.cc:1397] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:44:29.363112 7780 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: No bootstrap required, opened a new log
I20260926 05:44:29.363183 7780 ts_tablet_manager.cc:1397] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:44:29.364006 7778 raft_consensus.cc:348] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.364172 7778 raft_consensus.cc:374] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:29.364216 7778 raft_consensus.cc:729] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7aae84afdf4c0e8d66cdb17d0515f2, State: Initialized, Role: FOLLOWER
I20260926 05:44:29.364355 7778 consensus_queue.cc:260] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.364638 7778 ts_tablet_manager.cc:1428] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:44:29.364604 7779 raft_consensus.cc:348] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.364749 7761 heartbeater.cc:503] Master 127.23.213.254:43631 was elected leader, sending a full tablet report...
I20260926 05:44:29.364780 7779 raft_consensus.cc:374] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:29.364823 7779 raft_consensus.cc:729] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659784a0e78748f7a1ce628fbd3f5440, State: Initialized, Role: FOLLOWER
I20260926 05:44:29.364964 7779 consensus_queue.cc:260] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.365231 7779 ts_tablet_manager.cc:1428] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:44:29.365198 7780 raft_consensus.cc:348] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.365365 7780 raft_consensus.cc:374] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:29.365360 7493 heartbeater.cc:503] Master 127.23.213.254:43631 was elected leader, sending a full tablet report...
I20260926 05:44:29.365411 7780 raft_consensus.cc:729] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1173c69126f41879a100eaefc15dae7, State: Initialized, Role: FOLLOWER
I20260926 05:44:29.365522 7780 consensus_queue.cc:260] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.365854 7780 ts_tablet_manager.cc:1428] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:44:29.365947 7627 heartbeater.cc:503] Master 127.23.213.254:43631 was elected leader, sending a full tablet report...
W20260926 05:44:29.397607 7628 tablet.cc:2369] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:44:29.492856 7494 tablet.cc:2369] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:44:29.573855 7762 tablet.cc:2369] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:44:29.597519 7786 raft_consensus.cc:482] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:29.597645 7786 raft_consensus.cc:504] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.598047 7786 leader_election.cc:288] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193:39559), 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903)
I20260926 05:44:29.601996 7775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.598397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:54041 (local address 127.23.213.195:34903)
0926 05:44:29.598455 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.598457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.598647 (+ 190us) server_negotiation.cc:407] Connection header received
0926 05:44:29.598694 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.598695 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.598706 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.598765 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:29.598903 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.599596 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.601021 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.601617 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.601681 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.601689 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.601882 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.601923 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 05:44:29.601996 7777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.598318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:41763 (local address 127.23.213.193:39559)
0926 05:44:29.598375 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.598377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.598410 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:44:29.598448 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.598449 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.598464 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.598531 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:29.598715 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.599514 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.601021 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.601580 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.601673 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.601683 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.601881 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.601923 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 05:44:29.602007 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.598337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:34903 (local address 127.23.213.194:54041)
0926 05:44:29.598542 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.598552 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:29.598611 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:29.598780 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:29.598780 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:29.598782 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:29.598835 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.598836 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.599713 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.599714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.600864 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.600867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.601687 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.601687 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.601793 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.601797 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.601857 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.601902 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"mutex_wait_us":20,"thread_start_us":124,"threads_started":1}
I20260926 05:44:29.602007 7629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.598228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:39559 (local address 127.23.213.194:41763)
0926 05:44:29.598290 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.598302 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:29.598355 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:29.598548 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:29.598549 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:29.598552 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:29.598630 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.598632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.599604 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.599604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.600864 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:29.600867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.601686 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.601687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:29.601797 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.601798 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:44:29.601856 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 05:44:29.601902 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":13}
I20260926 05:44:29.602429 7715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad9c7f7d604d778cbe9b7ff0dda379" candidate_uuid: "b1173c69126f41879a100eaefc15dae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" is_pre_election: true
I20260926 05:44:29.602550 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad9c7f7d604d778cbe9b7ff0dda379" candidate_uuid: "b1173c69126f41879a100eaefc15dae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659784a0e78748f7a1ce628fbd3f5440" is_pre_election: true
I20260926 05:44:29.602596 7715 raft_consensus.cc:2454] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1173c69126f41879a100eaefc15dae7 in term 0.
I20260926 05:44:29.602681 7447 raft_consensus.cc:2454] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1173c69126f41879a100eaefc15dae7 in term 0.
I20260926 05:44:29.602821 7515 leader_election.cc:302] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7aae84afdf4c0e8d66cdb17d0515f2, b1173c69126f41879a100eaefc15dae7; no voters:
I20260926 05:44:29.602959 7786 raft_consensus.cc:2781] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:44:29.603016 7786 raft_consensus.cc:482] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:44:29.603056 7786 raft_consensus.cc:3037] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:29.603735 7786 raft_consensus.cc:504] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.603870 7786 leader_election.cc:288] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [CANDIDATE]: Term 1 election: Requested vote from peers 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193:39559), 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903)
I20260926 05:44:29.604048 7715 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad9c7f7d604d778cbe9b7ff0dda379" candidate_uuid: "b1173c69126f41879a100eaefc15dae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2"
I20260926 05:44:29.604063 7447 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00ad9c7f7d604d778cbe9b7ff0dda379" candidate_uuid: "b1173c69126f41879a100eaefc15dae7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "659784a0e78748f7a1ce628fbd3f5440"
I20260926 05:44:29.604127 7715 raft_consensus.cc:3037] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:29.604151 7447 raft_consensus.cc:3037] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:29.604830 7715 raft_consensus.cc:2454] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1173c69126f41879a100eaefc15dae7 in term 1.
I20260926 05:44:29.604985 7515 leader_election.cc:302] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d7aae84afdf4c0e8d66cdb17d0515f2, b1173c69126f41879a100eaefc15dae7; no voters:
I20260926 05:44:29.605028 7447 raft_consensus.cc:2454] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1173c69126f41879a100eaefc15dae7 in term 1.
I20260926 05:44:29.605108 7786 raft_consensus.cc:2781] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:29.605263 7786 raft_consensus.cc:686] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 1 LEADER]: Becoming Leader. State: Replica: b1173c69126f41879a100eaefc15dae7, State: Running, Role: LEADER
I20260926 05:44:29.605368 7786 consensus_queue.cc:237] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:29.606318 7300 catalog_manager.cc:5495] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1173c69126f41879a100eaefc15dae7 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "b1173c69126f41879a100eaefc15dae7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } health_report { overall_health: HEALTHY } } }
I20260926 05:44:29.637516 7776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.635262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.23.213.194:34749)
0926 05:44:29.635330 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:29.635332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:29.635347 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:44:29.635383 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:29.635384 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:29.635403 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:29.635463 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:44:29.635598 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.636071 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.636790 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:29.636970 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:29.637023 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:29.637166 (+ 143us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:44:29.637267 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:44:29.637327 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:29.637364 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 05:44:29.637382 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:44:29.639451 7447 raft_consensus.cc:1261] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Refusing update from remote peer b1173c69126f41879a100eaefc15dae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:29.639488 7715 raft_consensus.cc:1261] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Refusing update from remote peer b1173c69126f41879a100eaefc15dae7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:29.639814 7788 consensus_queue.cc:1035] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:29.639966 7786 consensus_queue.cc:1035] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:29.642453 7791 mvcc.cc:205] Tried to move back new op lower bound from 7333484419640336384 to 7333484419504209920. Current Snapshot: MvccSnapshot[applied={T|T < 7333484419640336384}]
I20260926 05:44:29.644552 7302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:29.644652 7302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42570:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:29.645380 7302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Sending DeleteTablet for 3 replicas of tablet 00ad9c7f7d604d778cbe9b7ff0dda379
I20260926 05:44:29.645689 7427 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ad9c7f7d604d778cbe9b7ff0dda379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:29 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:43184
I20260926 05:44:29.645727 7695 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ad9c7f7d604d778cbe9b7ff0dda379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:29 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:33560
I20260926 05:44:29.645764 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 7364
I20260926 05:44:29.645836 7561 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ad9c7f7d604d778cbe9b7ff0dda379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:29 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:40396
I20260926 05:44:29.647544 7800 tablet_replica.cc:330] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: stopping tablet replica
I20260926 05:44:29.647658 7800 raft_consensus.cc:2229] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:29.647766 7800 raft_consensus.cc:2258] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:29.648214 7800 ts_tablet_manager.cc:1905] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:29.649890 7801 tablet_replica.cc:330] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: stopping tablet replica
I20260926 05:44:29.650040 7801 raft_consensus.cc:2229] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:44:29.650194 7801 raft_consensus.cc:2258] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:29.650223 7800 ts_tablet_manager.cc:1918] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:29.650306 7800 log.cc:1196] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/00ad9c7f7d604d778cbe9b7ff0dda379
I20260926 05:44:29.650593 7800 ts_tablet_manager.cc:1939] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Deleting consensus metadata
I20260926 05:44:29.650975 7801 ts_tablet_manager.cc:1905] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:29.651014 7287 catalog_manager.cc:4841] TS 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903): tablet 00ad9c7f7d604d778cbe9b7ff0dda379 (table test-table [id=908a71f0496b4a238d9ea2ec033f0061]) successfully deleted
W20260926 05:44:29.652189 7289 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:39559 (error 108)
I20260926 05:44:29.652474 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 7498
I20260926 05:44:29.652694 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.652581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:39559 (local address 127.0.0.1:43188)
0926 05:44:29.652626 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.652657 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 05:44:29.652806 7289 catalog_manager.cc:4584] TS 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193:39559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 00ad9c7f7d604d778cbe9b7ff0dda379: Network error: Client connection negotiation failed: client connection to 127.23.213.193:39559: connect: Connection refused (error 111)
I20260926 05:44:29.652850 7801 ts_tablet_manager.cc:1918] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:29.657505 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 7632
I20260926 05:44:29.658267 7774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:29.657539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:34749 (local address 127.0.0.1:40414)
0926 05:44:29.658197 (+ 658us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:29.658228 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.194:34749: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":636}
I20260926 05:44:29.662205 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 7270
I20260926 05:44:49.667675 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43631
--webserver_interface=127.23.213.254
--webserver_port=42997
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:44:49.751766 7803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:49.751981 7803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:49.752007 7803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:49.753084 7803 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:44:49.753122 7803 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:44:49.753139 7803 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:44:49.753151 7803 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:44:49.753172 7803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:49.753190 7803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:49.753219 7803 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:44:49.753234 7803 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:44:49.754684 7803 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:43631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:49.754884 7803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:49.755064 7803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:49.764158 7809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:49.764303 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:49.764187 7810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:49.764422 7803 server_base.cc:1034] running on GCE node
I20260926 05:44:49.764669 7803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:49.764964 7803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:49.766122 7803 hybrid_clock.cc:648] HybridClock initialized: now 1790401489766106 us; error 25 us; skew 500 ppm
Sep 26 05:44:49 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:49.768816 7803 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:44:49.769373 7803 webserver.cc:460] Webserver started at http://127.23.213.254:42997/ using document root <none> and password file <none>
I20260926 05:44:49.769645 7803 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:49.769727 7803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:49.771256 7803 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:49.771962 7819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:49.772125 7803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:49.772210 7803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "38aa5f402ab4424981662b7716c49a29"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "2c0f75ee734ee0c93890c78a10248625"
server_key_iv: "f5e23dad9381859e9825d325925f9b6e"
server_key_version: "encryptionkey@0"
I20260926 05:44:49.772680 7803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:49.783234 7803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:49.785076 7803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:49.785193 7803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:49.789172 7803 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:43631
I20260926 05:44:49.789233 7871 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:43631 every 8 connection(s)
I20260926 05:44:49.789649 7803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:44:49.791174 7872 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:44:49.791777 7872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Bootstrap starting.
I20260926 05:44:49.792797 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 7803
I20260926 05:44:49.793079 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:39559
--local_ip_for_outbound_sockets=127.23.213.193
--tserver_master_addrs=127.23.213.254:43631
--webserver_port=42775
--webserver_interface=127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:49.800707 7872 log.cc:824] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:49.802749 7872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:44:49.803083 7872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Bootstrap complete.
I20260926 05:44:49.805343 7872 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:49.805567 7872 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38aa5f402ab4424981662b7716c49a29, State: Initialized, Role: FOLLOWER
I20260926 05:44:49.805775 7872 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:49.805856 7872 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:44:49.805918 7872 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:44:49.805967 7872 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 1 FOLLOWER]: Advancing to term 2
I20260926 05:44:49.807030 7872 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:49.807137 7872 leader_election.cc:302] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38aa5f402ab4424981662b7716c49a29; no voters:
I20260926 05:44:49.807350 7872 leader_election.cc:288] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 05:44:49.807394 7876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 05:44:49.807555 7876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 LEADER]: Becoming Leader. State: Replica: 38aa5f402ab4424981662b7716c49a29, State: Running, Role: LEADER
I20260926 05:44:49.807621 7872 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:44:49.807691 7876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:44:49.807994 7877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "38aa5f402ab4424981662b7716c49a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } } }
I20260926 05:44:49.808173 7877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:49.807996 7876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38aa5f402ab4424981662b7716c49a29. Latest consensus state: current_term: 2 leader_uuid: "38aa5f402ab4424981662b7716c49a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } } }
I20260926 05:44:49.808271 7876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: This master's current role is: LEADER
I20260926 05:44:49.808614 7885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:44:49.809484 7885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:44:49.809599 7885 catalog_manager.cc:1259] Loaded cluster ID: d7d27a95c5b74035bac17b5bf7383021
I20260926 05:44:49.809641 7885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:44:49.810690 7885 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:44:49.810892 7885 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Loaded TSK: 0
I20260926 05:44:49.824277 7885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Generated new TSK 1
I20260926 05:44:49.824404 7885 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 05:44:49.887683 7874 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:49.887951 7874 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:49.887992 7874 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:49.889127 7874 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:49.889189 7874 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:49.889233 7874 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:44:49.890722 7874 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=42775
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7874
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:49.890966 7874 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:49.891201 7874 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:49.897080 7900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:49.897060 7899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:49.897328 7874 server_base.cc:1034] running on GCE node
W20260926 05:44:49.897388 7902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:49.897624 7874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:49.897888 7874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:49.899065 7874 hybrid_clock.cc:648] HybridClock initialized: now 1790401489899050 us; error 33 us; skew 500 ppm
Sep 26 05:44:49 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:49.901710 7874 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:44:49.902256 7874 webserver.cc:460] Webserver started at http://127.23.213.193:42775/ using document root <none> and password file <none>
I20260926 05:44:49.902521 7874 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:49.902602 7874 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:49.904321 7874 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260926 05:44:49.905076 7909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:49.905279 7874 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:49.905376 7874 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "659784a0e78748f7a1ce628fbd3f5440"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "d79192cb49ecd28beefc5182d8b75457"
server_key_iv: "5c456bb8fc924d68106e6cc9566bfd18"
server_key_version: "encryptionkey@0"
I20260926 05:44:49.905829 7874 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:49.921444 7874 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:49.923947 7874 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:49.924104 7874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:49.924355 7874 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:49.924826 7916 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:44:49.926337 7874 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:44:49.926399 7874 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:44:49.926435 7874 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:44:49.927287 7874 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:44:49.927333 7874 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:49.927399 7916 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap starting.
I20260926 05:44:49.934293 7874 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:39559
I20260926 05:44:49.934468 8023 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:39559 every 8 connection(s)
I20260926 05:44:49.934729 7874 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:44:49.937983 7916 log.cc:824] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:49.939282 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 7874
I20260926 05:44:49.939626 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:34749
--local_ip_for_outbound_sockets=127.23.213.194
--tserver_master_addrs=127.23.213.254:43631
--webserver_port=36257
--webserver_interface=127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:44:49.940918 7916 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:44:49.941253 7916 tablet_bootstrap.cc:492] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap complete.
I20260926 05:44:49.942597 7916 ts_tablet_manager.cc:1397] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260926 05:44:49.944375 7916 raft_consensus.cc:348] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:49.944759 7916 raft_consensus.cc:729] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659784a0e78748f7a1ce628fbd3f5440, State: Initialized, Role: FOLLOWER
I20260926 05:44:49.944944 7916 consensus_queue.cc:260] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } } peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } }
I20260926 05:44:49.945251 7916 ts_tablet_manager.cc:1428] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
Sep 26 05:44:49 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:49.951800 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:49.935525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:45641)
0926 05:44:49.935722 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:49.935770 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:49.936356 (+ 586us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:49.938360 (+ 2004us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:49.938365 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:49.938747 (+ 382us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:49.939038 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:49.939044 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.942994 (+ 3950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.942996 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:49.944243 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:49.944247 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.945214 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.945215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:49.945350 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:49.946033 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:49.946043 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:49.947740 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:49.950068 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:49.950074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:49.950076 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:49.950347 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:49.950613 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:49.950614 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:49.950614 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:49.950736 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:49.951190 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:49.951196 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:49.951332 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 05:44:49.951427 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 05:44:49.951588 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":103,"threads_started":1}
I20260926 05:44:49.952845 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:49.935623 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:45641 (local address 127.23.213.254:43631)
0926 05:44:49.937863 (+ 2240us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:49.937874 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:49.937913 (+ 39us) server_negotiation.cc:407] Connection header received
0926 05:44:49.938046 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:49.938054 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:49.938112 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:49.941206 (+ 3094us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:49.941302 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.942851 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.944348 (+ 1497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.945116 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.945674 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:49.947859 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:49.947863 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:49.947864 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:49.947874 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:49.949979 (+ 2105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:49.950432 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:49.950435 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:49.950436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:49.950545 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:49.950804 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:49.950804 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:49.950805 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:49.951092 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:49.951155 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:49.952394 (+ 1239us) server_negotiation.cc:299] Negotiation successful
0926 05:44:49.952633 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2167,"thread_start_us":67,"threads_started":1}
I20260926 05:44:49.954082 8024 heartbeater.cc:347] Connected to a master server at 127.23.213.254:43631
I20260926 05:44:49.954175 8024 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:49.954358 8024 heartbeater.cc:511] Master 127.23.213.254:43631 requested a full tablet report, sending...
I20260926 05:44:49.955098 7836 ts_manager.cc:194] Registered new tserver with Master: 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193:39559)
I20260926 05:44:49.956352 7836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:45641
I20260926 05:44:49.957376 8024 heartbeater.cc:503] Master 127.23.213.254:43631 was elected leader, sending a full tablet report...
Sep 26 05:44:49 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401489, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.193@KRBTEST.COM
I20260926 05:44:49.965826 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:49.955717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:39559 (local address 127.0.0.1:46106)
0926 05:44:49.956141 (+ 424us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:49.956179 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:49.956248 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:49.956718 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:49.956721 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:49.957043 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:49.957099 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:49.957101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.958529 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.958530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:49.959428 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:49.959430 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.960118 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.960119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:49.960199 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:49.960714 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:49.960719 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:49.962529 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:49.964609 (+ 2080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:49.964615 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:49.964617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:49.964872 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:49.965089 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:49.965089 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:49.965090 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:49.965157 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:49.965468 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:49.965474 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:49.965524 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 05:44:49.965604 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:44:49.965709 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":87,"threads_started":1}
I20260926 05:44:49.966198 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:49.955969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46106 (local address 127.23.213.193:39559)
0926 05:44:49.956380 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:49.956384 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:49.956395 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:44:49.956504 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:49.956506 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:49.956523 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:49.956585 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:49.957182 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.958440 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.959526 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:49.959992 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:49.960057 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:49.962636 (+ 2579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:49.962644 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:49.962644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:49.962652 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:49.964519 (+ 1867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:49.964953 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:49.964956 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:49.964957 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:49.965019 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:49.965227 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:49.965228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:49.965228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:49.965400 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:49.965479 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:49.965655 (+ 176us) server_negotiation.cc:299] Negotiation successful
0926 05:44:49.965767 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":75,"threads_started":1}
I20260926 05:44:49.966514 7958 tablet_service.cc:1514] Processing DeleteTablet for tablet 00ad9c7f7d604d778cbe9b7ff0dda379 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:29 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:46106
I20260926 05:44:49.967182 8033 tablet_replica.cc:330] stopping tablet replica
I20260926 05:44:49.967307 8033 raft_consensus.cc:2229] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:49.967401 8033 raft_consensus.cc:2258] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:49.967718 8033 ts_tablet_manager.cc:1905] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:49.969902 8033 ts_tablet_manager.cc:1918] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:49.969986 8033 log.cc:1196] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal/wals/00ad9c7f7d604d778cbe9b7ff0dda379
I20260926 05:44:49.970203 8033 ts_tablet_manager.cc:1939] T 00ad9c7f7d604d778cbe9b7ff0dda379 P 659784a0e78748f7a1ce628fbd3f5440: Deleting consensus metadata
I20260926 05:44:49.970662 7823 catalog_manager.cc:4841] TS 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193:39559): tablet 00ad9c7f7d604d778cbe9b7ff0dda379 (table test-table [id=908a71f0496b4a238d9ea2ec033f0061]) successfully deleted
W20260926 05:44:50.027036 8029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:50.027231 8029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:50.027254 8029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:50.028404 8029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:50.028445 8029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:50.028471 8029 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:44:50.029938 8029 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:34749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=36257
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:50.030196 8029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:50.030429 8029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:50.035586 8039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:50.035861 8029 server_base.cc:1034] running on GCE node
W20260926 05:44:50.035573 8041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:50.035560 8038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:50.036252 8029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:50.036583 8029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:50.037777 8029 hybrid_clock.cc:648] HybridClock initialized: now 1790401490037721 us; error 41 us; skew 500 ppm
Sep 26 05:44:50 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:50.040622 8029 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:44:50.041087 8029 webserver.cc:460] Webserver started at http://127.23.213.194:36257/ using document root <none> and password file <none>
I20260926 05:44:50.041327 8029 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:50.041414 8029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:50.042980 8029 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:50.043651 8048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:50.043839 8029 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:50.043944 8029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "b1173c69126f41879a100eaefc15dae7"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "2fef76be80067e765856af9c264ac456"
server_key_iv: "6f2e6f3a16f5d74ab5e4b4a6ee3d00ea"
server_key_version: "encryptionkey@0"
I20260926 05:44:50.044310 8029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:50.061518 8029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:50.063565 8029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:50.063746 8029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:50.063992 8029 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:50.064433 8055 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
W20260926 05:44:50.065796 8055 ts_tablet_manager.cc:1852] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260926 05:44:50.066064 8055 ts_tablet_manager.cc:1905] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:50.068373 8055 ts_tablet_manager.cc:1918] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:50.068449 8055 log.cc:1196] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/00ad9c7f7d604d778cbe9b7ff0dda379
I20260926 05:44:50.068714 8055 ts_tablet_manager.cc:1939] T 00ad9c7f7d604d778cbe9b7ff0dda379 P b1173c69126f41879a100eaefc15dae7: Deleting consensus metadata
I20260926 05:44:50.068915 8029 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260926 05:44:50.068974 8029 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.005s user 0.000s sys 0.000s
I20260926 05:44:50.069020 8029 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:50.069051 8029 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:50.074896 8029 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:34749
I20260926 05:44:50.074951 8162 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:34749 every 8 connection(s)
I20260926 05:44:50.075284 8029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:44:50.076363 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8029
I20260926 05:44:50.076649 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:34903
--local_ip_for_outbound_sockets=127.23.213.195
--tserver_master_addrs=127.23.213.254:43631
--webserver_port=45509
--webserver_interface=127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 05:44:50 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:50.084322 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.075954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:38083 (local address 127.23.213.254:43631)
0926 05:44:50.076003 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.076005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.076292 (+ 287us) server_negotiation.cc:407] Connection header received
0926 05:44:50.076815 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.076817 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.076836 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.076903 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:50.077523 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.078102 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.079103 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.079315 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.079387 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.081505 (+ 2118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:50.081507 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:50.081508 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:50.081516 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:50.082974 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.083378 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.083380 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.083380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.083436 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.083677 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.083678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.083678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.083791 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:50.083863 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.084137 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.084228 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:44:50.084555 8165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.075966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:38083)
0926 05:44:50.076127 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:50.076177 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:50.076709 (+ 532us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:50.076980 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:50.076984 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:50.077233 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:50.077424 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.077429 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.078194 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.078195 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.079011 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.079013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.079406 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.079407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.079527 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.080095 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:50.080104 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:50.081412 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:50.083066 (+ 1654us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:50.083073 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:50.083075 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:50.083293 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:50.083500 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:50.083501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:50.083501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:50.083608 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:50.083867 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:50.083874 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:50.084009 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 05:44:50.084115 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 05:44:50.084269 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":85,"threads_started":1}
I20260926 05:44:50.084993 8163 heartbeater.cc:347] Connected to a master server at 127.23.213.254:43631
I20260926 05:44:50.085112 8163 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:50.085325 8163 heartbeater.cc:511] Master 127.23.213.254:43631 requested a full tablet report, sending...
I20260926 05:44:50.085726 7836 ts_manager.cc:194] Registered new tserver with Master: b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
I20260926 05:44:50.086122 7836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:38083
W20260926 05:44:50.157385 8166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:44:50.157582 8166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:44:50.157605 8166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:44:50.158794 8166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:44:50.158835 8166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:44:50.158895 8166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:44:50.160956 8166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:34903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=45509
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8166
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:44:50.161204 8166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:44:50.161435 8166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:44:50.167322 8172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:50.167343 8171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:44:50.167322 8174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:44:50.168567 8166 server_base.cc:1034] running on GCE node
I20260926 05:44:50.168809 8166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:44:50.169049 8166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:44:50.170200 8166 hybrid_clock.cc:648] HybridClock initialized: now 1790401490170186 us; error 26 us; skew 500 ppm
Sep 26 05:44:50 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:44:50.172721 8166 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:44:50.173254 8166 webserver.cc:460] Webserver started at http://127.23.213.195:45509/ using document root <none> and password file <none>
I20260926 05:44:50.173502 8166 fs_manager.cc:359] Metadata directory not provided
I20260926 05:44:50.173581 8166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:44:50.175033 8166 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:44:50.175662 8181 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:50.175833 8166 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:44:50.175935 8166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "91ba731fdedb5130452f60cac627a828"
server_key_iv: "82a7d3cc4a318be0d859f07b0e0b8029"
server_key_version: "encryptionkey@0"
I20260926 05:44:50.176287 8166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:44:50.187666 8166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:44:50.189976 8166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:44:50.190125 8166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:44:50.190351 8166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:44:50.190635 8166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:44:50.190683 8166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:50.190724 8166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:44:50.190762 8166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:44:50.196740 8166 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:34903
I20260926 05:44:50.196784 8294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:34903 every 8 connection(s)
I20260926 05:44:50.197114 8166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:44:50.201479 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8166
Sep 26 05:44:50 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401490, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:44:50.207144 8027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.197772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:57139 (local address 127.23.213.254:43631)
0926 05:44:50.197844 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.197847 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.198142 (+ 295us) server_negotiation.cc:407] Connection header received
0926 05:44:50.198640 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.198641 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.198650 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.198696 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:50.199609 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.200086 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.201054 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.201236 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.201344 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.203852 (+ 2508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:50.203853 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:50.203854 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:50.203861 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:50.205466 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.205980 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.205982 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.205982 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.206041 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.206390 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.206391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.206392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.206525 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:50.206597 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.206946 (+ 349us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.207057 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:44:50.207293 8297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.197724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:57139)
0926 05:44:50.197972 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:50.198027 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:50.198556 (+ 529us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:50.198756 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:50.198759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:50.199320 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:44:50.199537 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.199542 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.200156 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.200156 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.200984 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.200986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.201361 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.201362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.201464 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.202078 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:44:50.202088 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:44:50.203772 (+ 1684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:44:50.205536 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:50.205543 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:50.205545 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:50.205910 (+ 365us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:50.206108 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:44:50.206108 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:44:50.206109 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:44:50.206252 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:44:50.206599 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:44:50.206605 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:44:50.206779 (+ 174us) client_negotiation.cc:769] Sending connection context
0926 05:44:50.206875 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 05:44:50.206997 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"thread_start_us":92,"threads_started":1}
I20260926 05:44:50.207805 8295 heartbeater.cc:347] Connected to a master server at 127.23.213.254:43631
I20260926 05:44:50.207901 8295 heartbeater.cc:464] Registering TS with master...
I20260926 05:44:50.208118 8295 heartbeater.cc:511] Master 127.23.213.254:43631 requested a full tablet report, sending...
I20260926 05:44:50.208547 7836 ts_manager.cc:194] Registered new tserver with Master: 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903)
I20260926 05:44:50.209174 7836 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:57139
I20260926 05:44:50.210094 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.201949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60038 (local address 127.23.213.254:43631)
0926 05:44:50.202069 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.202071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.202783 (+ 712us) server_negotiation.cc:407] Connection header received
0926 05:44:50.202850 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.202850 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.202859 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.202910 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:50.203649 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.204313 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.205059 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.205390 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.205467 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.206772 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:50.206773 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:50.206774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:50.206779 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:50.208809 (+ 2030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.209207 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.209208 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.209208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.209320 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.209532 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.209533 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.209533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.209661 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:50.209861 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.209925 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.210005 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
I20260926 05:44:50.210816 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:44:50.213274 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.211110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60050 (local address 127.23.213.254:43631)
0926 05:44:50.211164 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.211166 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.211286 (+ 120us) server_negotiation.cc:407] Connection header received
0926 05:44:50.211333 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.211334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.211362 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.211427 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:44:50.211505 (+ 78us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.211988 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.212663 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.212882 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.212943 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.213060 (+ 117us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:44:50.213099 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 05:44:50.213220 (+ 121us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60050: token expired
Metrics: {"server-negotiator.queue_time_us":34}
W20260926 05:44:50.213348 8299 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60050: token expired
W20260926 05:44:50.213908 7764 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.23.213.254:43631)
I20260926 05:44:50.220731 8299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.214172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60054 (local address 127.23.213.254:43631)
0926 05:44:50.214213 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.214215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.214226 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:44:50.214250 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.214252 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.214262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.214331 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:44:50.214801 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.215526 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.216453 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.216776 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.216874 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.218095 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:44:50.218096 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:44:50.218097 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:44:50.218103 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:44:50.219761 (+ 1658us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.220061 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.220062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.220062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.220132 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:44:50.220282 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:44:50.220282 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:44:50.220283 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:44:50.220411 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:44:50.220514 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.220601 (+ 87us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.220675 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:44:50.221851 7836 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:44:50.223434 7836 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:44:50.225785 7958 tablet_service.cc:1467] Processing CreateTablet for tablet 6d627f457bda497f9a6621f6c6ec6ed8 (DEFAULT_TABLE table=test-table [id=9ff363634aee40a290fbb82936c41ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:50.226013 7958 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d627f457bda497f9a6621f6c6ec6ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:50.227715 7916 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap starting.
I20260926 05:44:50.228154 7916 tablet_bootstrap.cc:654] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:50.228744 7916 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: No bootstrap required, opened a new log
I20260926 05:44:50.228796 7916 ts_tablet_manager.cc:1397] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260926 05:44:50.229199 7916 raft_consensus.cc:348] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.229283 7916 raft_consensus.cc:374] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:50.229326 7916 raft_consensus.cc:729] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659784a0e78748f7a1ce628fbd3f5440, State: Initialized, Role: FOLLOWER
I20260926 05:44:50.229342 8301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.225393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:34903 (local address 127.0.0.1:55766)
0926 05:44:50.225550 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:50.225561 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:50.225608 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:50.226106 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:50.226107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:50.226109 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:50.226153 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.226154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.227096 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.227097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.227991 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.227993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.229027 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.229029 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.229126 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.229151 (+ 25us) client_negotiation.cc:769] Sending connection context
0926 05:44:50.229223 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:44:50.229265 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":6,"thread_start_us":98,"threads_started":1}
I20260926 05:44:50.229367 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.225631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.23.213.195:34903)
0926 05:44:50.225833 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.225838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.225854 (+ 16us) server_negotiation.cc:407] Connection header received
0926 05:44:50.225907 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.225909 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.225927 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.226005 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:50.226233 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.227014 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.228081 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.228931 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.229016 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.229089 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.229244 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.229286 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":79,"threads_started":1}
I20260926 05:44:50.229383 7916 consensus_queue.cc:260] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.229504 7916 ts_tablet_manager.cc:1428] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 05:44:50.229960 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.225386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:34749 (local address 127.0.0.1:52118)
0926 05:44:50.225423 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:50.225434 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:50.225469 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:50.226093 (+ 624us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:50.226095 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:50.226096 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:50.226144 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.226146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.227500 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.227501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.228558 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.228561 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.229684 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.229685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.229781 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.229785 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:50.229851 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:44:50.229894 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 05:44:50.229984 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.225471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52118 (local address 127.23.213.194:34749)
0926 05:44:50.225890 (+ 419us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.225895 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.225906 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:44:50.225946 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.225948 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.225975 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.226223 (+ 248us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:50.226310 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.227393 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.228722 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.229595 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.229683 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.229734 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.229866 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.229906 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":88,"threads_started":1}
I20260926 05:44:50.230302 8229 tablet_service.cc:1467] Processing CreateTablet for tablet 6d627f457bda497f9a6621f6c6ec6ed8 (DEFAULT_TABLE table=test-table [id=9ff363634aee40a290fbb82936c41ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:50.230635 8229 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d627f457bda497f9a6621f6c6ec6ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:50.233319 8305 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Bootstrap starting.
I20260926 05:44:50.233906 8097 tablet_service.cc:1467] Processing CreateTablet for tablet 6d627f457bda497f9a6621f6c6ec6ed8 (DEFAULT_TABLE table=test-table [id=9ff363634aee40a290fbb82936c41ef7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:44:50.234035 8305 tablet_bootstrap.cc:654] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:50.234208 8097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6d627f457bda497f9a6621f6c6ec6ed8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:44:50.234519 8305 log.cc:824] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:50.235251 8305 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: No bootstrap required, opened a new log
I20260926 05:44:50.235304 8305 ts_tablet_manager.cc:1397] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:44:50.236292 8055 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Bootstrap starting.
I20260926 05:44:50.236959 8055 tablet_bootstrap.cc:654] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Neither blocks nor log segments found. Creating new log.
I20260926 05:44:50.236936 8305 raft_consensus.cc:348] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.237102 8305 raft_consensus.cc:374] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:50.237169 8305 raft_consensus.cc:729] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d7aae84afdf4c0e8d66cdb17d0515f2, State: Initialized, Role: FOLLOWER
I20260926 05:44:50.237274 8055 log.cc:824] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Log is configured to *not* fsync() on all Append() calls
I20260926 05:44:50.237294 8305 consensus_queue.cc:260] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.237583 8295 heartbeater.cc:503] Master 127.23.213.254:43631 was elected leader, sending a full tablet report...
I20260926 05:44:50.237584 8305 ts_tablet_manager.cc:1428] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:44:50.238075 8055 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: No bootstrap required, opened a new log
I20260926 05:44:50.238144 8055 ts_tablet_manager.cc:1397] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:44:50.239887 8055 raft_consensus.cc:348] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.240031 8055 raft_consensus.cc:374] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:44:50.240077 8055 raft_consensus.cc:729] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1173c69126f41879a100eaefc15dae7, State: Initialized, Role: FOLLOWER
I20260926 05:44:50.240193 8055 consensus_queue.cc:260] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.240448 8055 ts_tablet_manager.cc:1428] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:44:50.240480 8163 heartbeater.cc:503] Master 127.23.213.254:43631 was elected leader, sending a full tablet report...
I20260926 05:44:50.269963 8030 raft_consensus.cc:482] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:44:50.270102 8030 raft_consensus.cc:504] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.270514 8030 leader_election.cc:288] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903), b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
I20260926 05:44:50.273504 8026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.270711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:34903 (local address 127.23.213.193:60239)
0926 05:44:50.270792 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:50.270804 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:50.270861 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:50.271040 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:50.271041 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:50.271043 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:50.271129 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.271131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.271750 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.271751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.272525 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.272527 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.273256 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.273257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.273323 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.273347 (+ 24us) client_negotiation.cc:769] Sending connection context
0926 05:44:50.273395 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:44:50.273430 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59}
I20260926 05:44:50.273533 8303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.270762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:60239 (local address 127.23.213.195:34903)
0926 05:44:50.270827 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.270829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.270840 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:44:50.270956 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.270957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.270969 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.271036 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:50.271199 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.271669 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.272625 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.273177 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.273248 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.273257 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.273422 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.273465 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:44:50.273947 8310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.270724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:34749 (local address 127.23.213.193:36443)
0926 05:44:50.271003 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:50.271013 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:44:50.271099 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:44:50.271280 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:44:50.271281 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:44:50.271282 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:44:50.271359 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.271361 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.272020 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.272021 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.272851 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:44:50.272854 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.273629 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.273631 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:44:50.273728 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.273732 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:44:50.273812 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 05:44:50.273850 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":263,"mutex_wait_us":91,"thread_start_us":140,"threads_started":1}
I20260926 05:44:50.273905 8249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d627f457bda497f9a6621f6c6ec6ed8" candidate_uuid: "659784a0e78748f7a1ce628fbd3f5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" is_pre_election: true
I20260926 05:44:50.274060 8249 raft_consensus.cc:2454] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659784a0e78748f7a1ce628fbd3f5440 in term 0.
I20260926 05:44:50.274286 7911 leader_election.cc:302] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659784a0e78748f7a1ce628fbd3f5440, 8d7aae84afdf4c0e8d66cdb17d0515f2; no voters:
I20260926 05:44:50.274307 8302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.270833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:36443 (local address 127.23.213.194:34749)
0926 05:44:50.270900 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.270903 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.271082 (+ 179us) server_negotiation.cc:407] Connection header received
0926 05:44:50.271155 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.271156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.271166 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.271234 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:44:50.271434 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.271952 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.272944 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.273516 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.273585 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.273595 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.274196 (+ 601us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.274250 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:44:50.274425 8030 raft_consensus.cc:2781] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:44:50.274482 8030 raft_consensus.cc:482] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:44:50.274513 8030 raft_consensus.cc:3037] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:50.274595 8117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d627f457bda497f9a6621f6c6ec6ed8" candidate_uuid: "659784a0e78748f7a1ce628fbd3f5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1173c69126f41879a100eaefc15dae7" is_pre_election: true
I20260926 05:44:50.274756 8117 raft_consensus.cc:2454] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 659784a0e78748f7a1ce628fbd3f5440 in term 0.
I20260926 05:44:50.275300 8030 raft_consensus.cc:504] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.275473 8030 leader_election.cc:288] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 1 election: Requested vote from peers 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903), b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
I20260926 05:44:50.275643 8249 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d627f457bda497f9a6621f6c6ec6ed8" candidate_uuid: "659784a0e78748f7a1ce628fbd3f5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2"
I20260926 05:44:50.275655 8117 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6d627f457bda497f9a6621f6c6ec6ed8" candidate_uuid: "659784a0e78748f7a1ce628fbd3f5440" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1173c69126f41879a100eaefc15dae7"
I20260926 05:44:50.275719 8249 raft_consensus.cc:3037] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:50.275743 8117 raft_consensus.cc:3037] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:44:50.276513 8249 raft_consensus.cc:2454] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659784a0e78748f7a1ce628fbd3f5440 in term 1.
I20260926 05:44:50.276523 8117 raft_consensus.cc:2454] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 659784a0e78748f7a1ce628fbd3f5440 in term 1.
I20260926 05:44:50.276688 7911 leader_election.cc:302] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 659784a0e78748f7a1ce628fbd3f5440, 8d7aae84afdf4c0e8d66cdb17d0515f2; no voters:
I20260926 05:44:50.276784 8030 raft_consensus.cc:2781] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:44:50.276921 8030 raft_consensus.cc:686] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 LEADER]: Becoming Leader. State: Replica: 659784a0e78748f7a1ce628fbd3f5440, State: Running, Role: LEADER
I20260926 05:44:50.277019 8030 consensus_queue.cc:237] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:44:50.277894 7836 catalog_manager.cc:5495] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 reported cstate change: term changed from 0 to 1, leader changed from <none> to 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "659784a0e78748f7a1ce628fbd3f5440" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } health_report { overall_health: HEALTHY } } }
I20260926 05:44:50.288944 8032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.286948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.23.213.193:39559)
0926 05:44:50.286992 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:44:50.286994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:44:50.287001 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:44:50.287072 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:44:50.287073 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:44:50.287089 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:44:50.287142 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:44:50.287253 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.287719 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.288364 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:44:50.288548 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:44:50.288596 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:44:50.288707 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:44:50.288800 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:44:50.288860 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 05:44:50.288877 (+ 17us) server_negotiation.cc:299] Negotiation successful
0926 05:44:50.288893 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:44:50.290402 8117 raft_consensus.cc:1261] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 1 FOLLOWER]: Refusing update from remote peer 659784a0e78748f7a1ce628fbd3f5440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:50.290397 8249 raft_consensus.cc:1261] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Refusing update from remote peer 659784a0e78748f7a1ce628fbd3f5440: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:44:50.290733 8311 consensus_queue.cc:1035] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:50.290859 8030 consensus_queue.cc:1035] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:44:50.294142 8314 mvcc.cc:205] Tried to move back new op lower bound from 7333484504227094528 to 7333484504175501312. Current Snapshot: MvccSnapshot[applied={T|T < 7333484504227094528 or (T in {7333484504227094528})}]
I20260926 05:44:50.295842 7835 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:50.295933 7835 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:44:50.296612 7835 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Sending DeleteTablet for 3 replicas of tablet 6d627f457bda497f9a6621f6c6ec6ed8
I20260926 05:44:50.296932 7958 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d627f457bda497f9a6621f6c6ec6ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:50 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:46106
I20260926 05:44:50.296934 8097 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d627f457bda497f9a6621f6c6ec6ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:50 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:52118
I20260926 05:44:50.297012 8229 tablet_service.cc:1514] Processing DeleteTablet for tablet 6d627f457bda497f9a6621f6c6ec6ed8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:44:50 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:55766
I20260926 05:44:50.297197 8324 tablet_replica.cc:330] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: stopping tablet replica
I20260926 05:44:50.297195 8033 tablet_replica.cc:330] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: stopping tablet replica
I20260926 05:44:50.297264 8033 raft_consensus.cc:2229] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:44:50.297304 8324 raft_consensus.cc:2229] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:50.297356 8033 raft_consensus.cc:2258] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:50.297413 8324 raft_consensus.cc:2258] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:50.297490 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 7874
I20260926 05:44:50.298484 8324 ts_tablet_manager.cc:1905] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:50.300490 8324 ts_tablet_manager.cc:1918] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:50.300565 8324 log.cc:1196] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/6d627f457bda497f9a6621f6c6ec6ed8
I20260926 05:44:50.300698 8325 tablet_replica.cc:330] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: stopping tablet replica
I20260926 05:44:50.300815 8325 raft_consensus.cc:2229] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:44:50.300829 8324 ts_tablet_manager.cc:1939] T 6d627f457bda497f9a6621f6c6ec6ed8 P b1173c69126f41879a100eaefc15dae7: Deleting consensus metadata
I20260926 05:44:50.300942 8325 raft_consensus.cc:2258] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:44:50.301393 8325 ts_tablet_manager.cc:1905] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:44:50.301393 7820 catalog_manager.cc:4841] TS b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749): tablet 6d627f457bda497f9a6621f6c6ec6ed8 (table test-table [id=9ff363634aee40a290fbb82936c41ef7]) successfully deleted
I20260926 05:44:50.302961 8325 ts_tablet_manager.cc:1918] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:44:50.303020 8325 log.cc:1196] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/6d627f457bda497f9a6621f6c6ec6ed8
I20260926 05:44:50.303341 8325 ts_tablet_manager.cc:1939] T 6d627f457bda497f9a6621f6c6ec6ed8 P 8d7aae84afdf4c0e8d66cdb17d0515f2: Deleting consensus metadata
W20260926 05:44:50.303877 7823 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:39559 (error 108)
I20260926 05:44:50.304019 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 8029
I20260926 05:44:50.304152 7821 catalog_manager.cc:4841] TS 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903): tablet 6d627f457bda497f9a6621f6c6ec6ed8 (table test-table [id=9ff363634aee40a290fbb82936c41ef7]) successfully deleted
I20260926 05:44:50.304829 8031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:44:50.304690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:39559 (local address 127.0.0.1:46132)
0926 05:44:50.304751 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 05:44:50.304790 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 05:44:50.304975 7823 catalog_manager.cc:4584] TS 659784a0e78748f7a1ce628fbd3f5440 (127.23.213.193:39559): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6d627f457bda497f9a6621f6c6ec6ed8: Network error: Client connection negotiation failed: client connection to 127.23.213.193:39559: connect: Connection refused (error 111)
I20260926 05:44:50.309486 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 8166
I20260926 05:44:50.316349 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 7803
I20260926 05:45:10.322232 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43631
--webserver_interface=127.23.213.254
--webserver_port=42997
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:43631
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:10.403417 8327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:10.403609 8327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:10.403630 8327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:10.404788 8327 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 05:45:10.404826 8327 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 05:45:10.404839 8327 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 05:45:10.404850 8327 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:10.404903 8327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:10.404922 8327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:10.404945 8327 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:10.404976 8327 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:10.406474 8327 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:43631
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=42997
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:10.406730 8327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:10.406966 8327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:10.411767 8333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:10.411789 8332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:10.411789 8335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:10.411952 8327 server_base.cc:1034] running on GCE node
I20260926 05:45:10.412182 8327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:10.412470 8327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:10.413614 8327 hybrid_clock.cc:648] HybridClock initialized: now 1790401510413596 us; error 33 us; skew 500 ppm
Sep 26 05:45:10 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:10.416079 8327 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:10.416558 8327 webserver.cc:460] Webserver started at http://127.23.213.254:42997/ using document root <none> and password file <none>
I20260926 05:45:10.416785 8327 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:10.416852 8327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:10.418284 8327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:10.419173 8342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.419436 8327 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:10.419528 8327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "38aa5f402ab4424981662b7716c49a29"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "2c0f75ee734ee0c93890c78a10248625"
server_key_iv: "f5e23dad9381859e9825d325925f9b6e"
server_key_version: "encryptionkey@0"
I20260926 05:45:10.420017 8327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:10.442670 8327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:10.445120 8327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:10.445268 8327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:10.449280 8327 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:43631
I20260926 05:45:10.449353 8394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:43631 every 8 connection(s)
I20260926 05:45:10.449733 8327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:10.451251 8395 sys_catalog.cc:263] Verifying existing consensus state
I20260926 05:45:10.451952 8395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Bootstrap starting.
I20260926 05:45:10.457512 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8327
I20260926 05:45:10.457829 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:39559
--local_ip_for_outbound_sockets=127.23.213.193
--tserver_master_addrs=127.23.213.254:43631
--webserver_port=42775
--webserver_interface=127.23.213.193
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:45:10.463119 8395 log.cc:824] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:10.465440 8395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:45:10.465811 8395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Bootstrap complete.
I20260926 05:45:10.468062 8395 raft_consensus.cc:348] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:45:10.468400 8395 raft_consensus.cc:729] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38aa5f402ab4424981662b7716c49a29, State: Initialized, Role: FOLLOWER
I20260926 05:45:10.468571 8395 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:45:10.468638 8395 raft_consensus.cc:388] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:10.468694 8395 raft_consensus.cc:482] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:10.468739 8395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 2 FOLLOWER]: Advancing to term 3
I20260926 05:45:10.469828 8395 raft_consensus.cc:504] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:45:10.469942 8395 leader_election.cc:302] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 38aa5f402ab4424981662b7716c49a29; no voters:
I20260926 05:45:10.470122 8395 leader_election.cc:288] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 05:45:10.470160 8399 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 05:45:10.470281 8399 raft_consensus.cc:686] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [term 3 LEADER]: Becoming Leader. State: Replica: 38aa5f402ab4424981662b7716c49a29, State: Running, Role: LEADER
I20260926 05:45:10.470382 8395 sys_catalog.cc:564] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:10.470374 8399 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } }
I20260926 05:45:10.470806 8401 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "38aa5f402ab4424981662b7716c49a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } } }
I20260926 05:45:10.470929 8401 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:10.470803 8399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38aa5f402ab4424981662b7716c49a29. Latest consensus state: current_term: 3 leader_uuid: "38aa5f402ab4424981662b7716c49a29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "38aa5f402ab4424981662b7716c49a29" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43631 } } }
I20260926 05:45:10.471117 8399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:10.471226 8406 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:10.472002 8406 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:10.472100 8406 catalog_manager.cc:1259] Loaded cluster ID: d7d27a95c5b74035bac17b5bf7383021
I20260926 05:45:10.472312 8406 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:10.473452 8406 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:10.473620 8406 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Loaded TSK: 0
I20260926 05:45:10.473667 8406 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 38aa5f402ab4424981662b7716c49a29: Loaded TSK: 1
F20260926 05:45:10.473915 8406 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb9e0fef9ad google::LogMessage::Fail() at ??:0
@ 0x7fb9e0ff288a google::LogMessage::SendToLog() at ??:0
@ 0x7fb9e0fef4de google::LogMessage::Flush() at ??:0
@ 0x7fb9e0ff0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb9e375c104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb9e3759d41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb9e8f491cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb9e8f4948e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb9e8f49594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7fb9e8f0fc48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fb9e8f19ad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb9e2ebd0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb9e2eb2954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb9e3d166db start_thread at ??:0
@ 0x7fb9dfaed61f clone at ??:0
W20260926 05:45:10.558945 8397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:10.559139 8397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:10.559162 8397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:10.560320 8397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:10.560364 8397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:10.560391 8397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:45:10.561944 8397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=42775
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:10.562175 8397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:10.562408 8397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:10.567570 8437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:10.567601 8436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:10.567574 8439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:10.567857 8397 server_base.cc:1034] running on GCE node
I20260926 05:45:10.568050 8397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:10.568349 8397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:10.569532 8397 hybrid_clock.cc:648] HybridClock initialized: now 1790401510569506 us; error 39 us; skew 500 ppm
Sep 26 05:45:10 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:10.572549 8397 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:45:10.573098 8397 webserver.cc:460] Webserver started at http://127.23.213.193:42775/ using document root <none> and password file <none>
I20260926 05:45:10.573351 8397 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:10.573421 8397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:10.575251 8397 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:10.576040 8446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.576283 8397 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:10.576390 8397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "659784a0e78748f7a1ce628fbd3f5440"
format_stamp: "Formatted at 2026-09-26 05:44:28 on dist-test-slave-1fdt"
server_key: "d79192cb49ecd28beefc5182d8b75457"
server_key_iv: "5c456bb8fc924d68106e6cc9566bfd18"
server_key_version: "encryptionkey@0"
I20260926 05:45:10.576822 8397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:10.597270 8397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:10.599764 8397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:10.599936 8397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:10.600219 8397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:10.600714 8453 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 05:45:10.602315 8397 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 05:45:10.602380 8397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 05:45:10.602416 8397 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 05:45:10.603336 8397 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 05:45:10.603390 8397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 05:45:10.603459 8453 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap starting.
I20260926 05:45:10.610661 8397 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:39559
I20260926 05:45:10.611109 8397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:45:10.611564 8560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:39559 every 8 connection(s)
I20260926 05:45:10.613731 8453 log.cc:824] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:10.614475 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8397
I20260926 05:45:10.614758 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:34749
--local_ip_for_outbound_sockets=127.23.213.194
--tserver_master_addrs=127.23.213.254:43631
--webserver_port=36257
--webserver_interface=127.23.213.194
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:45:10.616801 8453 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 05:45:10.617074 8453 tablet_bootstrap.cc:492] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Bootstrap complete.
I20260926 05:45:10.618113 8453 ts_tablet_manager.cc:1397] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
I20260926 05:45:10.619124 8453 raft_consensus.cc:348] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:45:10.619369 8453 raft_consensus.cc:729] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 659784a0e78748f7a1ce628fbd3f5440, State: Initialized, Role: FOLLOWER
I20260926 05:45:10.619539 8453 consensus_queue.cc:260] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:45:10.619804 8453 ts_tablet_manager.cc:1428] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
W20260926 05:45:10.679101 8563 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:45:10.611860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:45891)
0926 05:45:10.612070 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.612117 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:10.612695 (+ 578us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:10.678887 (+ 66192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":102,"threads_started":1}
W20260926 05:45:10.679414 8561 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:43631 (0 consecutive failures): Network error: Failed to ping master at 127.23.213.254:43631: Client connection negotiation failed: client connection to 127.23.213.254:43631: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 05:45:10.679775 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.679646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:56991)
0926 05:45:10.679703 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.679750 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:45:10.680116 8563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.680025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:42025)
0926 05:45:10.680072 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.680098 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:45:10.680245 8561 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:45:10.705783 8565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:10.705988 8565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:10.706014 8565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:10.707135 8565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:10.707173 8565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:10.707199 8565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:45:10.708576 8565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:34749
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=36257
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:10.708766 8565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:10.708949 8565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:10.714020 8574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:10.714092 8572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:10.714046 8571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:10.714184 8565 server_base.cc:1034] running on GCE node
I20260926 05:45:10.714443 8565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:10.714699 8565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:10.715894 8565 hybrid_clock.cc:648] HybridClock initialized: now 1790401510715885 us; error 35 us; skew 500 ppm
Sep 26 05:45:10 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:10.718441 8565 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:45:10.718906 8565 webserver.cc:460] Webserver started at http://127.23.213.194:36257/ using document root <none> and password file <none>
I20260926 05:45:10.719187 8565 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:10.719260 8565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:10.720593 8565 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:10.721212 8581 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.721390 8565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:45:10.721490 8565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "b1173c69126f41879a100eaefc15dae7"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "2fef76be80067e765856af9c264ac456"
server_key_iv: "6f2e6f3a16f5d74ab5e4b4a6ee3d00ea"
server_key_version: "encryptionkey@0"
I20260926 05:45:10.721894 8565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:10.733371 8565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:10.735188 8565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:10.735355 8565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:10.735579 8565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:10.735874 8565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:10.735929 8565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.735968 8565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:10.735996 8565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.742107 8565 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:34749
I20260926 05:45:10.742200 8694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:34749 every 8 connection(s)
I20260926 05:45:10.742532 8565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:45:10.742992 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8565
I20260926 05:45:10.743278 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:34903
--local_ip_for_outbound_sockets=127.23.213.195
--tserver_master_addrs=127.23.213.254:43631
--webserver_port=45509
--webserver_interface=127.23.213.195
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:43229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:45:10.743966 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.743251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:37971)
0926 05:45:10.743511 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.743592 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":113,"threads_started":1}
W20260926 05:45:10.744256 8695 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:43631 (0 consecutive failures): Network error: Failed to ping master at 127.23.213.254:43631: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
I20260926 05:45:10.744540 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.744444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:53373)
0926 05:45:10.744494 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.744518 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 05:45:10.744859 8697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.744776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:59787)
0926 05:45:10.744820 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.744843 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:45:10.744979 8695 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 05:45:10.826364 8698 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:10.826591 8698 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:10.826618 8698 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:10.827845 8698 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:10.827924 8698 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:10.827956 8698 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:45:10.829365 8698 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:34903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=45509
--tserver_master_addrs=127.23.213.254:43631
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8698
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:10.829550 8698 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:10.829792 8698 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:10.834929 8706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:10.835014 8698 server_base.cc:1034] running on GCE node
W20260926 05:45:10.834998 8703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:10.834950 8704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:10.835357 8698 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:10.835615 8698 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:10.836800 8698 hybrid_clock.cc:648] HybridClock initialized: now 1790401510836770 us; error 42 us; skew 500 ppm
Sep 26 05:45:10 dist-test-slave-1fdt krb5kdc[7257](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:10.839614 8698 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:45:10.840032 8698 webserver.cc:460] Webserver started at http://127.23.213.195:45509/ using document root <none> and password file <none>
I20260926 05:45:10.840260 8698 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:10.840330 8698 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:10.841641 8698 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:10.842296 8713 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.842476 8698 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:45:10.842578 8698 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2"
format_stamp: "Formatted at 2026-09-26 05:44:29 on dist-test-slave-1fdt"
server_key: "91ba731fdedb5130452f60cac627a828"
server_key_iv: "82a7d3cc4a318be0d859f07b0e0b8029"
server_key_version: "encryptionkey@0"
I20260926 05:45:10.842936 8698 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:10.851903 8698 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:10.853696 8698 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:10.853891 8698 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:10.854126 8698 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:10.854425 8698 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:10.854477 8698 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.854521 8698 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:10.854548 8698 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:10.860306 8698 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:34903
I20260926 05:45:10.860406 8826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:34903 every 8 connection(s)
I20260926 05:45:10.860733 8698 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:45:10.861871 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.861422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:50017)
0926 05:45:10.861589 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.861660 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
W20260926 05:45:10.862308 8827 heartbeater.cc:644] Failed to heartbeat to 127.23.213.254:43631 (0 consecutive failures): Network error: Failed to ping master at 127.23.213.254:43631: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
I20260926 05:45:10.862607 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.862518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:33735)
0926 05:45:10.862563 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.862590 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 05:45:10.862901 8829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:10.862815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:60265)
0926 05:45:10.862855 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:10.862888 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:45:10.863011 8827 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 05:45:10.868537 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8698
I20260926 05:45:11.680790 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:11.680545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:34677)
0926 05:45:11.680669 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:11.680724 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260926 05:45:11.745491 8832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:11.745263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:57793)
0926 05:45:11.745404 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:11.745439 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":90,"threads_started":1}
I20260926 05:45:11.863499 8833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:11.863307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:39569)
0926 05:45:11.863417 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:11.863451 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":57,"threads_started":1}
I20260926 05:45:11.982232 8834 raft_consensus.cc:482] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:11.982368 8834 raft_consensus.cc:504] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:45:11.982646 8834 leader_election.cc:288] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903), b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
Sep 26 05:45:11 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.194@KRBTEST.COM
Sep 26 05:45:11 dist-test-slave-1fdt krb5kdc[7257](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401510, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.195@KRBTEST.COM
I20260926 05:45:11.993266 8836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:11.982954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:49961 (local address 127.23.213.194:34749)
0926 05:45:11.983084 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:11.983098 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:11.983135 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:45:11.983724 (+ 589us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:11.983727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:11.983787 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:11.983944 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:11.984712 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.985728 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.986761 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.987158 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.987273 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:11.989650 (+ 2377us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:11.989654 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:11.989655 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:11.989667 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:11.991548 (+ 1881us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:11.992064 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:11.992067 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:11.992067 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:11.992170 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:11.992433 (+ 263us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:11.992434 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:11.992434 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:11.992652 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:11.992750 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:11.993017 (+ 267us) server_negotiation.cc:299] Negotiation successful
0926 05:45:11.993138 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":65,"threads_started":1}
I20260926 05:45:11.993254 8831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:11.982820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:34903 (local address 127.23.213.193:56779)
0926 05:45:11.982901 (+ 81us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:11.982911 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:11.982966 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:11.983991 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:11.983993 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:11.984378 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:11.984581 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:11.984587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.985917 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.985918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:11.986670 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:11.986672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.987257 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.987258 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:11.987393 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:11.987967 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:11.987975 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:11.989839 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:11.991744 (+ 1905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:11.991745 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:11.991745 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:11.991972 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:11.992222 (+ 250us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:11.992223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:11.992223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:11.992343 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:11.992717 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:11.992724 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:11.992855 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 05:45:11.992962 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 05:45:11.993150 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":36}
I20260926 05:45:11.993286 8837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:11.982992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:56779 (local address 127.23.213.195:34903)
0926 05:45:11.983142 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:11.983150 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:11.983178 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:45:11.983724 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:11.983727 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:11.983787 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:11.983943 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:11.984751 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.985821 (+ 1070us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.986762 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.987165 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.987281 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:11.989934 (+ 2653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:11.989937 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:11.989938 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:11.989947 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:11.991658 (+ 1711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:11.992059 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:11.992062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:11.992062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:11.992153 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:11.992411 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:11.992413 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:11.992413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:11.992646 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:11.992728 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:11.993029 (+ 301us) server_negotiation.cc:299] Negotiation successful
0926 05:45:11.993147 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
I20260926 05:45:11.994007 8835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:11.982820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:34749 (local address 127.23.213.193:49961)
0926 05:45:11.983045 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:11.983053 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:11.983089 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:11.984071 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:11.984072 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:11.984379 (+ 307us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:11.984581 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:11.984587 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.985870 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.985871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:11.986669 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:11.986671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:11.987258 (+ 587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:11.987259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:11.987388 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:11.988039 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:11.988043 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:11.989490 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:11.991633 (+ 2143us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:11.991640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:11.991642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:11.991972 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:11.992264 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:11.992265 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:11.992265 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:11.992347 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:11.992749 (+ 402us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:11.992750 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:11.992854 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 05:45:11.992962 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 05:45:11.993232 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"mutex_wait_us":184,"thread_start_us":73,"threads_started":1}
W20260926 05:45:11.994149 8448 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
W20260926 05:45:11.994719 8447 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
I20260926 05:45:11.994766 8447 leader_election.cc:302] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 659784a0e78748f7a1ce628fbd3f5440; no voters: 8d7aae84afdf4c0e8d66cdb17d0515f2, b1173c69126f41879a100eaefc15dae7
I20260926 05:45:11.994866 8834 raft_consensus.cc:2726] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:45:12.681577 8838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:12.681334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:55137)
0926 05:45:12.681493 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:12.681532 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260926 05:45:12.746161 8839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:12.745961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:49867)
0926 05:45:12.746088 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:12.746125 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":66,"threads_started":1}
I20260926 05:45:12.864151 8840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:12.863954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:33049)
0926 05:45:12.864075 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:12.864112 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260926 05:45:13.682376 8841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:13.682135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:44167)
0926 05:45:13.682285 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:13.682328 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":59,"threads_started":1}
I20260926 05:45:13.726511 8842 raft_consensus.cc:482] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:13.726601 8842 raft_consensus.cc:504] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:45:13.726805 8842 leader_election.cc:288] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903), b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
W20260926 05:45:13.727216 8447 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
W20260926 05:45:13.727303 8448 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
I20260926 05:45:13.727406 8448 leader_election.cc:302] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 659784a0e78748f7a1ce628fbd3f5440; no voters: 8d7aae84afdf4c0e8d66cdb17d0515f2, b1173c69126f41879a100eaefc15dae7
I20260926 05:45:13.727506 8842 raft_consensus.cc:2726] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:45:13.746891 8843 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:13.746575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:44927)
0926 05:45:13.746806 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:13.746849 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":187,"threads_started":1}
I20260926 05:45:13.864883 8844 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:13.864647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:44983)
0926 05:45:13.864803 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:13.864845 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260926 05:45:14.683079 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:14.682872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:37701)
0926 05:45:14.682993 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:14.683032 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260926 05:45:14.747476 8846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:14.747277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:39455)
0926 05:45:14.747402 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:14.747440 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260926 05:45:14.865650 8847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:14.865429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:43379)
0926 05:45:14.865575 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:14.865612 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":91,"threads_started":1}
I20260926 05:45:15.277618 8848 raft_consensus.cc:482] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:15.277722 8848 raft_consensus.cc:504] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:45:15.277953 8848 leader_election.cc:288] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903), b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
W20260926 05:45:15.278400 8448 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
W20260926 05:45:15.278483 8447 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
I20260926 05:45:15.278509 8447 leader_election.cc:302] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 659784a0e78748f7a1ce628fbd3f5440; no voters: 8d7aae84afdf4c0e8d66cdb17d0515f2, b1173c69126f41879a100eaefc15dae7
I20260926 05:45:15.278582 8848 raft_consensus.cc:2726] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:45:15.683801 8849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:15.683582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:38225)
0926 05:45:15.683716 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:15.683756 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":65,"threads_started":1}
I20260926 05:45:15.748178 8850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:15.747966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:48903)
0926 05:45:15.748095 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:15.748135 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260926 05:45:15.866302 8851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:15.866078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:33121)
0926 05:45:15.866192 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:15.866264 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260926 05:45:16.684530 8852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:16.684287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:43273)
0926 05:45:16.684421 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:16.684471 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260926 05:45:16.749012 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:16.748775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:46429)
0926 05:45:16.748909 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:16.748956 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":67,"threads_started":1}
I20260926 05:45:16.866995 8854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:16.866754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:34321)
0926 05:45:16.866910 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:16.866954 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":95,"threads_started":1}
I20260926 05:45:17.685189 8855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:17.684971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:45513)
0926 05:45:17.685105 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:17.685144 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260926 05:45:17.749699 8856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:17.749496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:39863)
0926 05:45:17.749624 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:17.749661 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260926 05:45:17.867594 8857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:17.867396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:54219)
0926 05:45:17.867527 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:17.867557 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":80,"threads_started":1}
I20260926 05:45:18.685871 8858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:18.685643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:58641)
0926 05:45:18.685778 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:18.685826 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":71,"threads_started":1}
I20260926 05:45:18.750401 8859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:18.750164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:42719)
0926 05:45:18.750305 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:18.750346 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":67,"threads_started":1}
I20260926 05:45:18.868441 8860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:18.868137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:39835)
0926 05:45:18.868331 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:18.868384 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":72,"threads_started":1}
I20260926 05:45:19.262560 8861 raft_consensus.cc:482] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:19.262657 8861 raft_consensus.cc:504] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d7aae84afdf4c0e8d66cdb17d0515f2" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 34903 } } peers { permanent_uuid: "b1173c69126f41879a100eaefc15dae7" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34749 } } peers { permanent_uuid: "659784a0e78748f7a1ce628fbd3f5440" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 39559 } }
I20260926 05:45:19.262831 8861 leader_election.cc:288] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903), b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749)
W20260926 05:45:19.263243 8448 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 8d7aae84afdf4c0e8d66cdb17d0515f2 (127.23.213.195:34903): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
W20260926 05:45:19.263301 8447 leader_election.cc:341] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b1173c69126f41879a100eaefc15dae7 (127.23.213.194:34749): Not found: Tablet not found: 6d627f457bda497f9a6621f6c6ec6ed8
I20260926 05:45:19.263336 8447 leader_election.cc:302] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 659784a0e78748f7a1ce628fbd3f5440; no voters: 8d7aae84afdf4c0e8d66cdb17d0515f2, b1173c69126f41879a100eaefc15dae7
I20260926 05:45:19.263417 8861 raft_consensus.cc:2726] T 6d627f457bda497f9a6621f6c6ec6ed8 P 659784a0e78748f7a1ce628fbd3f5440 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 05:45:19.686553 8862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:19.686336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:51273)
0926 05:45:19.686465 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:19.686506 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
I20260926 05:45:19.751113 8863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:19.750873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:58019)
0926 05:45:19.751002 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:19.751051 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260926 05:45:19.869285 8864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:19.869027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:41051)
0926 05:45:19.869184 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:19.869239 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":83,"threads_started":1}
I20260926 05:45:20.687359 8865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:20.687113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:54773)
0926 05:45:20.687249 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:20.687288 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":63,"threads_started":1}
I20260926 05:45:20.751866 8866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:20.751641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:41933)
0926 05:45:20.751781 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:20.751822 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":55,"threads_started":1}
I20260926 05:45:20.870100 8867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:20.869858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:46413)
0926 05:45:20.870009 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:20.870052 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":58,"threads_started":1}
I20260926 05:45:21.688139 8868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:21.687913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:37995)
0926 05:45:21.688053 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:21.688093 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":64,"threads_started":1}
I20260926 05:45:21.752635 8869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:21.752434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:46237)
0926 05:45:21.752552 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:21.752589 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260926 05:45:21.870795 8870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:21.870541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:60455)
0926 05:45:21.870706 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:21.870748 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":86,"threads_started":1}
I20260926 05:45:22.688932 8871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:22.688706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:37159)
0926 05:45:22.688845 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:22.688888 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":64,"threads_started":1}
I20260926 05:45:22.753487 8872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:22.753277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:34215)
0926 05:45:22.753406 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:22.753442 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":66,"threads_started":1}
I20260926 05:45:22.871443 8873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:22.871229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:54003)
0926 05:45:22.871358 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:22.871399 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":59,"threads_started":1}
I20260926 05:45:23.689659 8874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:23.689421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:33531)
0926 05:45:23.689545 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:23.689612 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":59,"threads_started":1}
I20260926 05:45:23.754197 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:23.753986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:41147)
0926 05:45:23.754113 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:23.754152 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":60,"threads_started":1}
I20260926 05:45:23.872021 8876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:23.871831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:58549)
0926 05:45:23.871948 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:23.871977 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":61,"threads_started":1}
I20260926 05:45:24.690346 8877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:24.690101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:43979)
0926 05:45:24.690259 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:24.690301 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260926 05:45:24.754866 8878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:24.754658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:32851)
0926 05:45:24.754781 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:24.754820 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260926 05:45:24.872696 8879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:24.872487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:52985)
0926 05:45:24.872605 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:24.872653 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":69,"threads_started":1}
I20260926 05:45:25.691154 8880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:25.690950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.193:54411)
0926 05:45:25.691071 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:25.691110 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260926 05:45:25.755648 8881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:25.755408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.194:46073)
0926 05:45:25.755558 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:25.755603 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":60,"threads_started":1}
I20260926 05:45:25.872285 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 8397
I20260926 05:45:25.873713 8882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:25.873446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:43631 (local address 127.23.213.195:38777)
0926 05:45:25.873587 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:25.873634 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.254:43631: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260926 05:45:25.878839 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 8565
I20260926 05:45:25.884883 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 8698
2026-09-26T05:45:25Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58378 ms)
[----------] 30 tests from SecurityITest (233717 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:25 dist-test-slave-1fdt krb5kdc[8885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:25 dist-test-slave-1fdt krb5kdc[8885](info): set up 2 sockets
Sep 26 05:45:25 dist-test-slave-1fdt krb5kdc[8885](info): commencing operation
krb5kdc: starting...
W20260926 05:45:26.933490 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:27.052867 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:45531
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:45531
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:27.133811 8898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:27.134068 8898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:27.134107 8898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:27.135286 8898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:27.135360 8898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:27.135385 8898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:27.135418 8898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:27.135442 8898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:27.136937 8898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:45531
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:45531
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:27.137214 8898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:27.137441 8898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:27.142405 8904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:27.142453 8898 server_base.cc:1034] running on GCE node
W20260926 05:45:27.142450 8903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:27.142421 8906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:27.142870 8898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:27.143164 8898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:27.144346 8898 hybrid_clock.cc:648] HybridClock initialized: now 1790401527144325 us; error 40 us; skew 500 ppm
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:27.147251 8898 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:27.147678 8898 webserver.cc:460] Webserver started at http://127.23.213.254:40883/ using document root <none> and password file <none>
I20260926 05:45:27.147921 8898 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:27.147990 8898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:27.148145 8898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:27.149215 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "598e2d9c9dc7454397d9bbe37e12be64"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "b95b2df7e9feb01b57332def5f30f13c"
server_key_iv: "e9891a737e226eec58cb32dc03dcaee5"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.149585 8898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "598e2d9c9dc7454397d9bbe37e12be64"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "b95b2df7e9feb01b57332def5f30f13c"
server_key_iv: "e9891a737e226eec58cb32dc03dcaee5"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.150913 8898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:27.151633 8913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.151814 8898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.151913 8898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "598e2d9c9dc7454397d9bbe37e12be64"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "b95b2df7e9feb01b57332def5f30f13c"
server_key_iv: "e9891a737e226eec58cb32dc03dcaee5"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.151997 8898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:27.165271 8898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:27.167766 8898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:27.167927 8898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:27.172003 8898 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:45531
I20260926 05:45:27.172036 8965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:45531 every 8 connection(s)
I20260926 05:45:27.172386 8898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:27.173218 8966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:27.175500 8966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64: Bootstrap starting.
I20260926 05:45:27.176084 8966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:27.176383 8966 log.cc:824] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:27.177055 8966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64: No bootstrap required, opened a new log
I20260926 05:45:27.177740 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8898
I20260926 05:45:27.177944 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:27.178177 24407 external_mini_cluster.cc:1428] Setting key 937107ddc3d49a317d1907c5751adb16
I20260926 05:45:27.179164 8966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598e2d9c9dc7454397d9bbe37e12be64" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45531 } }
I20260926 05:45:27.179350 8966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:27.179394 8966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 598e2d9c9dc7454397d9bbe37e12be64, State: Initialized, Role: FOLLOWER
I20260926 05:45:27.179528 8966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598e2d9c9dc7454397d9bbe37e12be64" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45531 } }
I20260926 05:45:27.179616 8966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:27.179649 8966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:27.179702 8966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:27.180641 8966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598e2d9c9dc7454397d9bbe37e12be64" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45531 } }
I20260926 05:45:27.180725 8966 leader_election.cc:302] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 598e2d9c9dc7454397d9bbe37e12be64; no voters:
I20260926 05:45:27.180918 8966 leader_election.cc:288] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:27.180938 8971 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:27.181049 8971 raft_consensus.cc:686] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [term 1 LEADER]: Becoming Leader. State: Replica: 598e2d9c9dc7454397d9bbe37e12be64, State: Running, Role: LEADER
I20260926 05:45:27.181129 8971 consensus_queue.cc:237] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598e2d9c9dc7454397d9bbe37e12be64" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45531 } }
I20260926 05:45:27.181177 8966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:27.181437 8971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "598e2d9c9dc7454397d9bbe37e12be64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598e2d9c9dc7454397d9bbe37e12be64" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45531 } } }
I20260926 05:45:27.181589 8971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:27.181816 8971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 598e2d9c9dc7454397d9bbe37e12be64. Latest consensus state: current_term: 1 leader_uuid: "598e2d9c9dc7454397d9bbe37e12be64" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "598e2d9c9dc7454397d9bbe37e12be64" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45531 } } }
I20260926 05:45:27.181946 8971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:27.182312 8979 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:27.182888 8979 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:27.184657 8979 catalog_manager.cc:1347] Generated new cluster ID: 6a989e61e6264495a1ad363392886530
I20260926 05:45:27.184715 8979 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:27.189141 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.178767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36994 (local address 127.23.213.254:45531)
0926 05:45:27.178949 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.178956 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.178981 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:45:27.179125 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.179131 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.179172 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.179279 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:27.180085 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.181202 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.182430 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.183072 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.183194 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.185902 (+ 2708us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:27.185905 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:27.185906 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:27.185918 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:27.187624 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.188006 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.188009 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.188009 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.188132 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.188350 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.188351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.188351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.188594 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:27.188764 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.188841 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.188984 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":86,"threads_started":1}
I20260926 05:45:27.209949 8979 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:27.210700 8979 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:27.223877 8979 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64: Generated new TSK 0
I20260926 05:45:27.224092 8979 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:45:27.260385 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:45531
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:27.341817 8992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:27.342043 8992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:27.342084 8992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:27.343277 8992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:27.343341 8992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:27.343384 8992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:45:27.344935 8992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:27.345197 8992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:27.345429 8992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:27.350462 8998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:27.350577 8992 server_base.cc:1034] running on GCE node
W20260926 05:45:27.350445 9000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:27.350445 8997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:27.350873 8992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:27.351068 8992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:27.352221 8992 hybrid_clock.cc:648] HybridClock initialized: now 1790401527352201 us; error 36 us; skew 500 ppm
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:27.354777 8992 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:45:27.355317 8992 webserver.cc:460] Webserver started at http://127.23.213.193:43613/ using document root <none> and password file <none>
I20260926 05:45:27.355587 8992 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:27.355631 8992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:27.355785 8992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:27.356803 8992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "c0fd85e0b519455bbf18aa4414858ad8"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "5f9dfaf3ce9ad12e4864d9d67624f85c"
server_key_iv: "25f3bde2dd8403e324ee5629ba84a05d"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.357112 8992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "c0fd85e0b519455bbf18aa4414858ad8"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "5f9dfaf3ce9ad12e4864d9d67624f85c"
server_key_iv: "25f3bde2dd8403e324ee5629ba84a05d"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.358474 8992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:27.359127 9007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.359320 8992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:27.359409 8992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "c0fd85e0b519455bbf18aa4414858ad8"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "5f9dfaf3ce9ad12e4864d9d67624f85c"
server_key_iv: "25f3bde2dd8403e324ee5629ba84a05d"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.359478 8992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:27.376367 8992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:27.378244 8992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:27.378401 8992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:27.378633 8992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:27.378973 8992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:27.379026 8992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.379067 8992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:27.379093 8992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.384929 8992 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:41907
I20260926 05:45:27.385015 9120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:41907 every 8 connection(s)
I20260926 05:45:27.385330 8992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:45:27.385596 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 8992
I20260926 05:45:27.385720 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:45:27.385970 24407 external_mini_cluster.cc:1428] Setting key 75b7d0d9e4b0fb04624ef3fc5c0ed276
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
I20260926 05:45:27.396697 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.386121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:47809 (local address 127.23.213.254:45531)
0926 05:45:27.386163 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.386165 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.386576 (+ 411us) server_negotiation.cc:407] Connection header received
0926 05:45:27.387361 (+ 785us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.387362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.387374 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.387421 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:27.388712 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.389229 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.390457 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.390694 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.390789 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.393283 (+ 2494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:27.393285 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:27.393285 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:27.393303 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:27.394908 (+ 1605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.395435 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.395436 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.395437 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.395522 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.395964 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.395965 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.395966 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.396178 (+ 212us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:27.396264 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.396545 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.396629 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:27.396957 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.386099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:45531 (local address 127.23.213.193:47809)
0926 05:45:27.386325 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.386427 (+ 102us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.387246 (+ 819us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.387500 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.387504 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.388377 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:27.388598 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.388604 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.389359 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.389360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.390348 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.390351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.390799 (+ 448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.390800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.390926 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.391521 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:27.391528 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:27.393159 (+ 1631us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:27.394984 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:27.394990 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:27.394992 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:27.395320 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:27.395633 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:27.395634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:27.395634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:27.395815 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:27.396280 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:27.396286 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:27.396438 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.396524 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.396641 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":94,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
I20260926 05:45:27.397470 9121 heartbeater.cc:347] Connected to a master server at 127.23.213.254:45531
I20260926 05:45:27.397572 9121 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:27.397787 9121 heartbeater.cc:511] Master 127.23.213.254:45531 requested a full tablet report, sending...
I20260926 05:45:27.398365 8930 ts_manager.cc:194] Registered new tserver with Master: c0fd85e0b519455bbf18aa4414858ad8 (127.23.213.193:41907)
I20260926 05:45:27.399019 8930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:47809
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:45:27.412623 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:45531
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:27.495460 9126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:27.495707 9126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:27.495746 9126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:27.496893 9126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:27.496963 9126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:27.497008 9126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:45:27.498538 9126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:27.498813 9126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:27.499045 9126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:27.504153 9134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:27.504123 9131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:27.504210 9132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:27.504470 9126 server_base.cc:1034] running on GCE node
I20260926 05:45:27.504714 9126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:27.504948 9126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:27.506094 9126 hybrid_clock.cc:648] HybridClock initialized: now 1790401527506091 us; error 54 us; skew 500 ppm
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:27.508669 9126 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:45:27.509106 9126 webserver.cc:460] Webserver started at http://127.23.213.194:42143/ using document root <none> and password file <none>
I20260926 05:45:27.509339 9126 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:27.509405 9126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:27.509557 9126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:27.510661 9126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "facdd80b6a944e4a82f76490a0e0bf59"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "376ce38f36c5262381887e2658fbb66e"
server_key_iv: "f01fa75c4e1ca6c6c1fe44658378c750"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.511037 9126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "facdd80b6a944e4a82f76490a0e0bf59"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "376ce38f36c5262381887e2658fbb66e"
server_key_iv: "f01fa75c4e1ca6c6c1fe44658378c750"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.512409 9126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:27.513172 9141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.513371 9126 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:27.513470 9126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "facdd80b6a944e4a82f76490a0e0bf59"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "376ce38f36c5262381887e2658fbb66e"
server_key_iv: "f01fa75c4e1ca6c6c1fe44658378c750"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.513558 9126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:27.540078 9126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:27.542562 9126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:27.542733 9126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:27.542990 9126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:27.543318 9126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:27.543371 9126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.543421 9126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:27.543444 9126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.549531 9126 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:46407
I20260926 05:45:27.549589 9254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:46407 every 8 connection(s)
I20260926 05:45:27.549970 9126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:27.557468 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 9126
I20260926 05:45:27.557585 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:45:27.557855 24407 external_mini_cluster.cc:1428] Setting key 1d46c9a51cef0c09aba2540c72d19c44
I20260926 05:45:27.559109 9257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.550700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:45531 (local address 127.23.213.194:59263)
0926 05:45:27.550885 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.550926 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.551408 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.551620 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.551623 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.552192 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:27.552397 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.552402 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.553042 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.553042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.553855 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.553856 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.554243 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.554244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.554351 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.554849 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:27.554858 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:27.556252 (+ 1394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:27.557661 (+ 1409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:27.557668 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:27.557670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:27.558010 (+ 340us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:27.558269 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:27.558269 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:27.558270 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:27.558369 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:27.558634 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:27.558639 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:27.558758 (+ 119us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.558841 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.558955 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":107,"threads_started":1}
I20260926 05:45:27.559108 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.550727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:59263 (local address 127.23.213.254:45531)
0926 05:45:27.550794 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.550796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.551014 (+ 218us) server_negotiation.cc:407] Connection header received
0926 05:45:27.551487 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.551488 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.551497 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.551562 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:27.552473 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.552967 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.553929 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.554166 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.554225 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.556344 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:27.556346 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:27.556346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:27.556353 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:27.557557 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.558102 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.558104 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.558104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.558194 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.558436 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.558437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.558437 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.558567 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:27.558643 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.558882 (+ 239us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.558969 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:45:27.559579 9255 heartbeater.cc:347] Connected to a master server at 127.23.213.254:45531
I20260926 05:45:27.559675 9255 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:27.559857 9255 heartbeater.cc:511] Master 127.23.213.254:45531 requested a full tablet report, sending...
I20260926 05:45:27.560298 8930 ts_manager.cc:194] Registered new tserver with Master: facdd80b6a944e4a82f76490a0e0bf59 (127.23.213.194:46407)
I20260926 05:45:27.560665 8930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:59263
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:45:27.585263 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:45531
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:27.665609 9260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:27.665876 9260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:27.665906 9260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:27.667016 9260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:27.667052 9260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:27.667078 9260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:45:27.668409 9260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:45531
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:27.668583 9260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:27.668769 9260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:27.673964 9265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:27.673964 9266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:27.673993 9268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:27.674079 9260 server_base.cc:1034] running on GCE node
I20260926 05:45:27.674431 9260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:27.674707 9260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:27.675868 9260 hybrid_clock.cc:648] HybridClock initialized: now 1790401527675851 us; error 32 us; skew 500 ppm
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:27.678490 9260 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:45:27.678925 9260 webserver.cc:460] Webserver started at http://127.23.213.195:32951/ using document root <none> and password file <none>
I20260926 05:45:27.679157 9260 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:27.679206 9260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:27.679373 9260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:27.680431 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "d9d8941199c343b89be084413c8f6ddb"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "41d4f5f1f430545ce31472389875b14d"
server_key_iv: "3bc966f7bd1547474cf93532ba410e54"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.680797 9260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "d9d8941199c343b89be084413c8f6ddb"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "41d4f5f1f430545ce31472389875b14d"
server_key_iv: "3bc966f7bd1547474cf93532ba410e54"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.682317 9260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:27.683128 9275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.683337 9260 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:27.683439 9260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "d9d8941199c343b89be084413c8f6ddb"
format_stamp: "Formatted at 2026-09-26 05:45:27 on dist-test-slave-1fdt"
server_key: "41d4f5f1f430545ce31472389875b14d"
server_key_iv: "3bc966f7bd1547474cf93532ba410e54"
server_key_version: "encryptionkey@0"
I20260926 05:45:27.683516 9260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:27.707973 9260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:27.710014 9260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:27.710175 9260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:27.710422 9260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:27.710731 9260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:27.710765 9260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.710788 9260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:27.710836 9260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:27.716761 9260 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:39393
I20260926 05:45:27.716804 9388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:39393 every 8 connection(s)
I20260926 05:45:27.717113 9260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:45:27.720171 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 9260
I20260926 05:45:27.720273 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:45:27.720463 24407 external_mini_cluster.cc:1428] Setting key 6bfedfdbde1a7e76c93e5812b25f9b67
Sep 26 05:45:27 dist-test-slave-1fdt krb5kdc[8885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401527, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:27.726581 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.717870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:52735 (local address 127.23.213.254:45531)
0926 05:45:27.717915 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.717917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.718153 (+ 236us) server_negotiation.cc:407] Connection header received
0926 05:45:27.718681 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.718682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.718694 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.718752 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:27.719686 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.720350 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.721651 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.721907 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.721963 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.724030 (+ 2067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:27.724031 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:27.724031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:27.724037 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:27.725428 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.725821 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.725822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.725822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.725873 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.726081 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.726082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.726082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.726175 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:27.726224 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.726448 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.726505 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:45:27.726737 9391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.717862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:45531 (local address 127.23.213.195:52735)
0926 05:45:27.718025 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.718059 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.718574 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.718822 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.718826 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.719394 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:27.719584 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.719589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.720435 (+ 846us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.720436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.721584 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.721587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.721972 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.721973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.722067 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.722492 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:27.722499 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:27.723940 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:27.725495 (+ 1555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:27.725500 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:27.725502 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:27.725703 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:27.725930 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:27.725930 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:27.725931 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:27.726027 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:27.726231 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:27.726235 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:27.726341 (+ 106us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.726422 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.726536 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":78,"threads_started":1}
I20260926 05:45:27.727180 9389 heartbeater.cc:347] Connected to a master server at 127.23.213.254:45531
I20260926 05:45:27.727290 9389 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:27.727461 9389 heartbeater.cc:511] Master 127.23.213.254:45531 requested a full tablet report, sending...
I20260926 05:45:27.727801 8930 ts_manager.cc:194] Registered new tserver with Master: d9d8941199c343b89be084413c8f6ddb (127.23.213.195:39393)
I20260926 05:45:27.728148 8930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:52735
I20260926 05:45:27.732157 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:45:27.738730 8970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.733407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38672 (local address 127.23.213.254:45531)
0926 05:45:27.733477 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.733479 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.733521 (+ 42us) server_negotiation.cc:407] Connection header received
0926 05:45:27.733573 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.733573 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.733581 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.733625 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:27.734202 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.734650 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.735318 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.735487 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.735533 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.736625 (+ 1092us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:27.736626 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:27.736626 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:27.736631 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:27.737800 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.738086 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.738087 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.738087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.738135 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:27.738281 (+ 146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:27.738282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:27.738282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:27.738410 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:27.738501 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.738579 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.738633 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 05:45:27.740176 8930 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38672:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:45:27.741729 8930 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:45:27.748138 9401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.744142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:46407 (local address 127.0.0.1:34756)
0926 05:45:27.744490 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.744500 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.744554 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.744822 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.744823 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.744825 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:27.744859 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.744860 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.746067 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.746068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.747169 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.747170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.747849 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.747850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.747929 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.747942 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.748015 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.748069 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":59,"threads_started":1}
I20260926 05:45:27.748149 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.744318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34756 (local address 127.23.213.194:46407)
0926 05:45:27.744476 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.744480 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.744543 (+ 63us) server_negotiation.cc:407] Connection header received
0926 05:45:27.744648 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.744650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.744665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.744740 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:27.744934 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.745967 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.747245 (+ 1278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.747759 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.747849 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.747905 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.748034 (+ 129us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.748071 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260926 05:45:27.748224 9404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.743960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41907 (local address 127.0.0.1:42200)
0926 05:45:27.744249 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.744288 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.744361 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.744615 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.744618 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.744620 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:27.744797 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.744799 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.745760 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.745761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.747021 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.747023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.748005 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.748006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.748069 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.748073 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.748121 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.748161 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":41,"threads_started":1}
I20260926 05:45:27.748560 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.744065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42200 (local address 127.23.213.193:41907)
0926 05:45:27.744263 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.744267 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.744368 (+ 101us) server_negotiation.cc:407] Connection header received
0926 05:45:27.744503 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.744506 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.744525 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.744596 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:27.744881 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.745647 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.747115 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.747845 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.747916 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.747968 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.748440 (+ 472us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.748486 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":82,"threads_started":1}
I20260926 05:45:27.748885 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.743955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:39393 (local address 127.0.0.1:50394)
0926 05:45:27.744237 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.744288 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.744359 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.744651 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.744651 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.744653 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:27.744773 (+ 120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.744775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.746239 (+ 1464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.746240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.747201 (+ 961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.747204 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.748123 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.748124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.748597 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.748677 (+ 80us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.748781 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.748823 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":257,"mutex_wait_us":61,"thread_start_us":85,"threads_started":1}
I20260926 05:45:27.748922 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.744031 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.23.213.195:39393)
0926 05:45:27.744188 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.744194 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.744343 (+ 149us) server_negotiation.cc:407] Connection header received
0926 05:45:27.744484 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.744487 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.744502 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.744553 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:27.744862 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.746114 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.747313 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.747984 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.748046 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.748092 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.748798 (+ 706us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.748842 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":84,"threads_started":1}
I20260926 05:45:27.748968 9189 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff57f79735a4ab3aee8b6a9995536cb (DEFAULT_TABLE table=test-table [id=6f050d33a49a4c96bfe227c748955bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:27.749192 9055 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff57f79735a4ab3aee8b6a9995536cb (DEFAULT_TABLE table=test-table [id=6f050d33a49a4c96bfe227c748955bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:27.749341 9189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff57f79735a4ab3aee8b6a9995536cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:27.749496 9055 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff57f79735a4ab3aee8b6a9995536cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:27.750433 9323 tablet_service.cc:1467] Processing CreateTablet for tablet 4ff57f79735a4ab3aee8b6a9995536cb (DEFAULT_TABLE table=test-table [id=6f050d33a49a4c96bfe227c748955bbc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:27.750727 9323 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ff57f79735a4ab3aee8b6a9995536cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:27.752740 9406 tablet_bootstrap.cc:492] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: Bootstrap starting.
I20260926 05:45:27.753506 9406 tablet_bootstrap.cc:654] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:27.753904 9407 tablet_bootstrap.cc:492] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Bootstrap starting.
I20260926 05:45:27.754127 9406 log.cc:824] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:27.754498 9407 tablet_bootstrap.cc:654] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:27.754805 9406 tablet_bootstrap.cc:492] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: No bootstrap required, opened a new log
I20260926 05:45:27.754858 9406 ts_tablet_manager.cc:1397] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:45:27.755020 9407 log.cc:824] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:27.755362 9408 tablet_bootstrap.cc:492] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Bootstrap starting.
I20260926 05:45:27.755861 9407 tablet_bootstrap.cc:492] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: No bootstrap required, opened a new log
I20260926 05:45:27.755930 9407 ts_tablet_manager.cc:1397] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:27.756664 9406 raft_consensus.cc:348] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.756800 9406 raft_consensus.cc:374] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:27.756824 9406 raft_consensus.cc:729] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0fd85e0b519455bbf18aa4414858ad8, State: Initialized, Role: FOLLOWER
I20260926 05:45:27.756935 9406 consensus_queue.cc:260] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.757254 9406 ts_tablet_manager.cc:1428] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:45:27.757278 9121 heartbeater.cc:503] Master 127.23.213.254:45531 was elected leader, sending a full tablet report...
I20260926 05:45:27.757656 9407 raft_consensus.cc:348] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.757882 9407 raft_consensus.cc:374] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:27.757930 9407 raft_consensus.cc:729] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: facdd80b6a944e4a82f76490a0e0bf59, State: Initialized, Role: FOLLOWER
I20260926 05:45:27.758041 9407 consensus_queue.cc:260] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.758322 9255 heartbeater.cc:503] Master 127.23.213.254:45531 was elected leader, sending a full tablet report...
I20260926 05:45:27.758307 9407 ts_tablet_manager.cc:1428] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 05:45:27.758859 9408 tablet_bootstrap.cc:654] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:27.759464 9408 log.cc:824] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:27.760084 9408 tablet_bootstrap.cc:492] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: No bootstrap required, opened a new log
I20260926 05:45:27.760135 9408 ts_tablet_manager.cc:1397] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260926 05:45:27.761397 9408 raft_consensus.cc:348] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.761539 9408 raft_consensus.cc:374] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:27.761559 9408 raft_consensus.cc:729] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d9d8941199c343b89be084413c8f6ddb, State: Initialized, Role: FOLLOWER
I20260926 05:45:27.761667 9408 consensus_queue.cc:260] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.761968 9408 ts_tablet_manager.cc:1428] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:45:27.762019 9389 heartbeater.cc:503] Master 127.23.213.254:45531 was elected leader, sending a full tablet report...
W20260926 05:45:27.800581 9256 tablet.cc:2369] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:27.864482 9411 raft_consensus.cc:482] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:27.864631 9411 raft_consensus.cc:504] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.865018 9411 leader_election.cc:288] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d9d8941199c343b89be084413c8f6ddb (127.23.213.195:39393), facdd80b6a944e4a82f76490a0e0bf59 (127.23.213.194:46407)
I20260926 05:45:27.868793 9405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.865416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:48983 (local address 127.23.213.194:46407)
0926 05:45:27.865475 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.865477 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.865585 (+ 108us) server_negotiation.cc:407] Connection header received
0926 05:45:27.865660 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.865661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.865673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.865728 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:27.865919 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.866581 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.867850 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.868423 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.868499 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.868506 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.868676 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.868713 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 05:45:27.868783 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.865307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:34365 (local address 127.23.213.195:39393)
0926 05:45:27.865353 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.865355 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.865365 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:45:27.865428 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.865429 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.865439 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.865498 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:27.865677 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.866466 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.867853 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.868423 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.868498 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.868506 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.868676 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.868712 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:45:27.868799 9123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.865214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:39393 (local address 127.23.213.193:34365)
0926 05:45:27.865277 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.865291 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.865357 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.865513 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.865515 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.865517 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:27.865602 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.865604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.866548 (+ 944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.866548 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.867751 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.867754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.868515 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.868516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.868597 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.868598 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.868657 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.868734 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":44}
I20260926 05:45:27.868806 9415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.865361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:46407 (local address 127.23.213.193:48983)
0926 05:45:27.865507 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.865517 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:27.865582 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:27.865786 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:27.865787 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:27.865788 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:27.865854 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.865855 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.866660 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.866661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.867751 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:27.867754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.868515 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.868516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:27.868593 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.868597 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:27.868657 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:45:27.868716 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"mutex_wait_us":2,"thread_start_us":78,"threads_started":1}
I20260926 05:45:27.869194 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff57f79735a4ab3aee8b6a9995536cb" candidate_uuid: "c0fd85e0b519455bbf18aa4414858ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d8941199c343b89be084413c8f6ddb" is_pre_election: true
I20260926 05:45:27.869292 9209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff57f79735a4ab3aee8b6a9995536cb" candidate_uuid: "c0fd85e0b519455bbf18aa4414858ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facdd80b6a944e4a82f76490a0e0bf59" is_pre_election: true
I20260926 05:45:27.869362 9343 raft_consensus.cc:2454] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd85e0b519455bbf18aa4414858ad8 in term 0.
I20260926 05:45:27.869405 9209 raft_consensus.cc:2454] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0fd85e0b519455bbf18aa4414858ad8 in term 0.
I20260926 05:45:27.869591 9008 leader_election.cc:302] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0fd85e0b519455bbf18aa4414858ad8, d9d8941199c343b89be084413c8f6ddb; no voters:
I20260926 05:45:27.869720 9411 raft_consensus.cc:2781] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:45:27.869838 9411 raft_consensus.cc:482] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:45:27.869866 9411 raft_consensus.cc:3037] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:27.870719 9411 raft_consensus.cc:504] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.870867 9411 leader_election.cc:288] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [CANDIDATE]: Term 1 election: Requested vote from peers d9d8941199c343b89be084413c8f6ddb (127.23.213.195:39393), facdd80b6a944e4a82f76490a0e0bf59 (127.23.213.194:46407)
I20260926 05:45:27.871030 9343 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff57f79735a4ab3aee8b6a9995536cb" candidate_uuid: "c0fd85e0b519455bbf18aa4414858ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d9d8941199c343b89be084413c8f6ddb"
I20260926 05:45:27.871049 9209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ff57f79735a4ab3aee8b6a9995536cb" candidate_uuid: "c0fd85e0b519455bbf18aa4414858ad8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "facdd80b6a944e4a82f76490a0e0bf59"
I20260926 05:45:27.871124 9209 raft_consensus.cc:3037] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:27.871126 9343 raft_consensus.cc:3037] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:27.871803 9343 raft_consensus.cc:2454] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd85e0b519455bbf18aa4414858ad8 in term 1.
I20260926 05:45:27.871950 9209 raft_consensus.cc:2454] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0fd85e0b519455bbf18aa4414858ad8 in term 1.
I20260926 05:45:27.871959 9008 leader_election.cc:302] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0fd85e0b519455bbf18aa4414858ad8, d9d8941199c343b89be084413c8f6ddb; no voters:
I20260926 05:45:27.872059 9411 raft_consensus.cc:2781] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:27.872208 9411 raft_consensus.cc:686] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 1 LEADER]: Becoming Leader. State: Replica: c0fd85e0b519455bbf18aa4414858ad8, State: Running, Role: LEADER
I20260926 05:45:27.872310 9411 consensus_queue.cc:237] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } }
I20260926 05:45:27.873116 8930 catalog_manager.cc:5495] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 reported cstate change: term changed from 0 to 1, leader changed from <none> to c0fd85e0b519455bbf18aa4414858ad8 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "c0fd85e0b519455bbf18aa4414858ad8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0fd85e0b519455bbf18aa4414858ad8" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41907 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 } health_report { overall_health: UNKNOWN } } }
W20260926 05:45:27.886124 9122 tablet.cc:2369] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:27.896638 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.894273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42216 (local address 127.23.213.193:41907)
0926 05:45:27.894331 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:27.894333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:27.894345 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:45:27.894390 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:27.894390 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:27.894414 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:27.894492 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:45:27.894612 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.895069 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.895834 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:27.896028 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:27.896100 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:27.896293 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:45:27.896387 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:45:27.896482 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:27.896506 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 05:45:27.896526 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:45:27.898404 9343 raft_consensus.cc:1261] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 1 FOLLOWER]: Refusing update from remote peer c0fd85e0b519455bbf18aa4414858ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:27.898434 9209 raft_consensus.cc:1261] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 1 FOLLOWER]: Refusing update from remote peer c0fd85e0b519455bbf18aa4414858ad8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:27.898798 9416 consensus_queue.cc:1035] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "facdd80b6a944e4a82f76490a0e0bf59" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 46407 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:27.898933 9411 consensus_queue.cc:1035] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d9d8941199c343b89be084413c8f6ddb" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 39393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:27.901525 9420 mvcc.cc:205] Tried to move back new op lower bound from 7333484658269339648 to 7333484658165788672. Current Snapshot: MvccSnapshot[applied={T|T < 7333484658269339648}]
I20260926 05:45:27.902459 9421 mvcc.cc:205] Tried to move back new op lower bound from 7333484658269339648 to 7333484658165788672. Current Snapshot: MvccSnapshot[applied={T|T < 7333484658269339648}]
I20260926 05:45:27.904109 8929 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:27.904210 8929 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38672:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:27.905000 8929 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 598e2d9c9dc7454397d9bbe37e12be64: Sending DeleteTablet for 3 replicas of tablet 4ff57f79735a4ab3aee8b6a9995536cb
I20260926 05:45:27.905365 9055 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff57f79735a4ab3aee8b6a9995536cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:27 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:42200
I20260926 05:45:27.905504 9189 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff57f79735a4ab3aee8b6a9995536cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:27 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:34756
I20260926 05:45:27.905514 9323 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ff57f79735a4ab3aee8b6a9995536cb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:27 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:50394
I20260926 05:45:27.905714 9429 tablet_replica.cc:330] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: stopping tablet replica
I20260926 05:45:27.905781 9428 tablet_replica.cc:330] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8: stopping tablet replica
I20260926 05:45:27.905836 9429 raft_consensus.cc:2229] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:27.905874 9428 raft_consensus.cc:2229] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:45:27.905941 9429 raft_consensus.cc:2258] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:27.905958 9428 raft_consensus.cc:2258] T 4ff57f79735a4ab3aee8b6a9995536cb P c0fd85e0b519455bbf18aa4414858ad8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:27.906004 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 8992
I20260926 05:45:27.907739 9429 ts_tablet_manager.cc:1905] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:27.909106 9430 tablet_replica.cc:330] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: stopping tablet replica
I20260926 05:45:27.909269 9430 raft_consensus.cc:2229] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:27.909344 9429 ts_tablet_manager.cc:1918] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:27.909402 9430 raft_consensus.cc:2258] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:27.909422 9429 log.cc:1196] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/4ff57f79735a4ab3aee8b6a9995536cb
I20260926 05:45:27.909713 9429 ts_tablet_manager.cc:1939] T 4ff57f79735a4ab3aee8b6a9995536cb P facdd80b6a944e4a82f76490a0e0bf59: Deleting consensus metadata
I20260926 05:45:27.909896 9430 ts_tablet_manager.cc:1905] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:27.910174 8914 catalog_manager.cc:4841] TS facdd80b6a944e4a82f76490a0e0bf59 (127.23.213.194:46407): tablet 4ff57f79735a4ab3aee8b6a9995536cb (table test-table [id=6f050d33a49a4c96bfe227c748955bbc]) successfully deleted
W20260926 05:45:27.911540 8916 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:41907 (error 108)
I20260926 05:45:27.911697 9430 ts_tablet_manager.cc:1918] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:27.911768 9430 log.cc:1196] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/4ff57f79735a4ab3aee8b6a9995536cb
I20260926 05:45:27.911801 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 9126
I20260926 05:45:27.912035 9430 ts_tablet_manager.cc:1939] T 4ff57f79735a4ab3aee8b6a9995536cb P d9d8941199c343b89be084413c8f6ddb: Deleting consensus metadata
I20260926 05:45:27.912444 8914 catalog_manager.cc:4841] TS d9d8941199c343b89be084413c8f6ddb (127.23.213.195:39393): tablet 4ff57f79735a4ab3aee8b6a9995536cb (table test-table [id=6f050d33a49a4c96bfe227c748955bbc]) successfully deleted
I20260926 05:45:27.912498 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:27.911920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41907 (local address 127.0.0.1:42222)
0926 05:45:27.912426 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:27.912469 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:41907: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481}
W20260926 05:45:27.912590 8916 catalog_manager.cc:4584] TS c0fd85e0b519455bbf18aa4414858ad8 (127.23.213.193:41907): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ff57f79735a4ab3aee8b6a9995536cb: Network error: Client connection negotiation failed: client connection to 127.23.213.193:41907: connect: Connection refused (error 111)
I20260926 05:45:27.916572 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 9260
I20260926 05:45:27.921727 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 8898
2026-09-26T05:45:27Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2036 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T05:45:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:27Z Disabled control of system clock
I20260926 05:45:27.949283 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:36575
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:36575
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:28.029644 9435 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:28.029942 9435 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:28.029963 9435 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:28.031086 9435 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:28.031118 9435 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:28.031131 9435 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:28.031144 9435 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:28.031157 9435 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:28.032404 9435 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:36575
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:36575
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9435
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:28.032588 9435 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:28.032826 9435 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:28.037492 9441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:28.037496 9443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:28.037503 9440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:28.037737 9435 server_base.cc:1034] running on GCE node
I20260926 05:45:28.037997 9435 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:28.038283 9435 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:28.039450 9435 hybrid_clock.cc:648] HybridClock initialized: now 1790401528039426 us; error 35 us; skew 500 ppm
I20260926 05:45:28.040659 9435 webserver.cc:460] Webserver started at http://127.23.213.254:42653/ using document root <none> and password file <none>
I20260926 05:45:28.040913 9435 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:28.040983 9435 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:28.041134 9435 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:28.042271 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "15aed23cad9d4f5e8a5dba2f319cf903"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "374ff905ea83254ed4d1f7b083fb0858"
server_key_iv: "0814a9c90c14b5a27a156222e11f0936"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.042662 9435 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "15aed23cad9d4f5e8a5dba2f319cf903"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "374ff905ea83254ed4d1f7b083fb0858"
server_key_iv: "0814a9c90c14b5a27a156222e11f0936"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.043943 9435 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:28.044660 9449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.044832 9435 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:28.044929 9435 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "15aed23cad9d4f5e8a5dba2f319cf903"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "374ff905ea83254ed4d1f7b083fb0858"
server_key_iv: "0814a9c90c14b5a27a156222e11f0936"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.045009 9435 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:28.069633 9435 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:28.070001 9435 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:28.070165 9435 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:28.074297 9435 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:36575
I20260926 05:45:28.074340 9501 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:36575 every 8 connection(s)
I20260926 05:45:28.074699 9435 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:28.075650 9502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:28.077924 9502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903: Bootstrap starting.
I20260926 05:45:28.078449 9502 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:28.078682 9502 log.cc:824] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:28.079310 9502 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903: No bootstrap required, opened a new log
I20260926 05:45:28.080776 9502 raft_consensus.cc:348] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 36575 } }
I20260926 05:45:28.080906 9502 raft_consensus.cc:374] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:28.080928 9502 raft_consensus.cc:729] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15aed23cad9d4f5e8a5dba2f319cf903, State: Initialized, Role: FOLLOWER
I20260926 05:45:28.081043 9502 consensus_queue.cc:260] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 36575 } }
I20260926 05:45:28.081095 9502 raft_consensus.cc:388] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:28.081116 9502 raft_consensus.cc:482] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:28.081142 9502 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:28.081595 9502 raft_consensus.cc:504] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 36575 } }
I20260926 05:45:28.081676 9502 leader_election.cc:302] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 15aed23cad9d4f5e8a5dba2f319cf903; no voters:
I20260926 05:45:28.081907 9502 leader_election.cc:288] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:28.081979 9505 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:28.082139 9505 raft_consensus.cc:686] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [term 1 LEADER]: Becoming Leader. State: Replica: 15aed23cad9d4f5e8a5dba2f319cf903, State: Running, Role: LEADER
I20260926 05:45:28.082181 9502 sys_catalog.cc:564] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:28.082264 9505 consensus_queue.cc:237] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 36575 } }
I20260926 05:45:28.082759 9506 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 36575 } } }
I20260926 05:45:28.082991 9507 sys_catalog.cc:455] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 15aed23cad9d4f5e8a5dba2f319cf903. Latest consensus state: current_term: 1 leader_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "15aed23cad9d4f5e8a5dba2f319cf903" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 36575 } } }
I20260926 05:45:28.083104 9507 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:28.083068 9506 sys_catalog.cc:458] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:28.083393 9516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:28.083832 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 9435
I20260926 05:45:28.083879 9516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:28.083954 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:28.084124 24407 external_mini_cluster.cc:1428] Setting key 1d65d32fc0a90f64fefbdd9aa9d12272
I20260926 05:45:28.089099 9516 catalog_manager.cc:1347] Generated new cluster ID: 663e44779eed4b73a7320678a61cb0c2
I20260926 05:45:28.089190 9516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:28.089632 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.084889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53608 (local address 127.23.213.254:36575)
0926 05:45:28.085051 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.085060 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.085092 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:45:28.085179 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.085184 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.085233 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.085341 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:28.085561 (+ 220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.086488 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.087612 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.088056 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.088146 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.088860 (+ 714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:28.088864 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:28.088864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:28.089066 (+ 202us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:28.089109 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:28.089124 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:28.089198 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.089340 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.089419 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":74,"threads_started":1}
I20260926 05:45:28.112890 9516 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:28.113451 9516 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:28.115752 9516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903: Generated new TSK 0
I20260926 05:45:28.115947 9516 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:28.120711 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:36575
--builtin_ntp_servers=127.23.213.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:28.206600 9526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:28.206812 9526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:28.206836 9526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:28.208017 9526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:28.208073 9526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:28.208101 9526 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:45:28.209622 9526 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:28.209899 9526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:28.210237 9526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:28.215355 9534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:28.215456 9526 server_base.cc:1034] running on GCE node
W20260926 05:45:28.215413 9531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:28.215560 9532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:28.215816 9526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:28.216125 9526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:28.217314 9526 hybrid_clock.cc:648] HybridClock initialized: now 1790401528217295 us; error 29 us; skew 500 ppm
I20260926 05:45:28.218655 9526 webserver.cc:460] Webserver started at http://127.23.213.193:44925/ using document root <none> and password file <none>
I20260926 05:45:28.218911 9526 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:28.218986 9526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:28.219149 9526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:28.220297 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "7764cc8e9e8444049b57f6ed50dd1828"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "5a2e84a7b2a5b8bba1af2698a4651522"
server_key_iv: "7ec35d75677a4352936a479142a68b45"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.220701 9526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "7764cc8e9e8444049b57f6ed50dd1828"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "5a2e84a7b2a5b8bba1af2698a4651522"
server_key_iv: "7ec35d75677a4352936a479142a68b45"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.222386 9526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:45:28.223094 9540 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.223248 9526 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:28.223351 9526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "7764cc8e9e8444049b57f6ed50dd1828"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "5a2e84a7b2a5b8bba1af2698a4651522"
server_key_iv: "7ec35d75677a4352936a479142a68b45"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.223430 9526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:28.250494 9526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:28.250831 9526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:28.251017 9526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:28.251279 9526 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:28.251621 9526 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:28.251675 9526 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.251715 9526 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:28.251740 9526 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.257589 9526 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:35955
I20260926 05:45:28.257645 9653 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:35955 every 8 connection(s)
I20260926 05:45:28.258009 9526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:45:28.262960 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.258755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:57189 (local address 127.23.213.254:36575)
0926 05:45:28.258836 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.258839 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.259046 (+ 207us) server_negotiation.cc:407] Connection header received
0926 05:45:28.259531 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.259532 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.259544 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.259595 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:28.260227 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.260713 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.261678 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.261942 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.261994 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.262730 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:28.262731 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:28.262732 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:28.262736 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:28.262750 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:28.262754 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:28.262816 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.262877 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.262902 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:45:28.263139 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.258710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:36575 (local address 127.23.213.193:57189)
0926 05:45:28.258921 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.258956 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.259448 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.259648 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.259652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.259660 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:28.260149 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.260153 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.260792 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.260793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.261602 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.261604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.262016 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.262016 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.262122 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.262602 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:28.262602 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:28.262629 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:28.262652 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:28.262815 (+ 163us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:28.262818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:28.262820 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.262866 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.262942 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":109,"threads_started":1}
I20260926 05:45:28.263659 9654 heartbeater.cc:347] Connected to a master server at 127.23.213.254:36575
I20260926 05:45:28.263777 9654 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:28.263976 9654 heartbeater.cc:511] Master 127.23.213.254:36575 requested a full tablet report, sending...
I20260926 05:45:28.264514 9466 ts_manager.cc:194] Registered new tserver with Master: 7764cc8e9e8444049b57f6ed50dd1828 (127.23.213.193:35955)
I20260926 05:45:28.265139 9466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.193:57189
I20260926 05:45:28.266801 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 9526
I20260926 05:45:28.266886 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:45:28.267016 24407 external_mini_cluster.cc:1428] Setting key 7004ae8d988f92918b850cb28e4f3f08
I20260926 05:45:28.267859 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:36575
--builtin_ntp_servers=127.23.213.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:28.348392 9657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:28.348626 9657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:28.348663 9657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:28.349836 9657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:28.349900 9657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:28.349939 9657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:45:28.351284 9657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:28.351557 9657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:28.351873 9657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:28.356773 9662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:28.356784 9665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:28.356784 9663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:28.356978 9657 server_base.cc:1034] running on GCE node
I20260926 05:45:28.357203 9657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:28.357467 9657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:28.358624 9657 hybrid_clock.cc:648] HybridClock initialized: now 1790401528358609 us; error 45 us; skew 500 ppm
I20260926 05:45:28.359823 9657 webserver.cc:460] Webserver started at http://127.23.213.194:33143/ using document root <none> and password file <none>
I20260926 05:45:28.360080 9657 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:28.360150 9657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:28.360308 9657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:28.361390 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "417fd07fd2914caaae12a83053341c6b"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "ae0ffed1ce302c54b445fb5df1c75494"
server_key_iv: "1ade82bd779a8220cf737bbb2a504909"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.361780 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "417fd07fd2914caaae12a83053341c6b"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "ae0ffed1ce302c54b445fb5df1c75494"
server_key_iv: "1ade82bd779a8220cf737bbb2a504909"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.363047 9657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:28.363754 9671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.363914 9657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:45:28.364001 9657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "417fd07fd2914caaae12a83053341c6b"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "ae0ffed1ce302c54b445fb5df1c75494"
server_key_iv: "1ade82bd779a8220cf737bbb2a504909"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.364082 9657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:28.380867 9657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:28.381186 9657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:28.381343 9657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:28.381578 9657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:28.381971 9657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:28.382027 9657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.382071 9657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:28.382097 9657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.388115 9657 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:33813
I20260926 05:45:28.388165 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:33813 every 8 connection(s)
I20260926 05:45:28.388489 9657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:45:28.392411 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 9657
I20260926 05:45:28.392529 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:45:28.392733 24407 external_mini_cluster.cc:1428] Setting key 8425d4fbe41a067e9e6fd177dbed7ebe
I20260926 05:45:28.393729 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:36575
--builtin_ntp_servers=127.23.213.212:43779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:45:28.393857 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.389180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:57069 (local address 127.23.213.254:36575)
0926 05:45:28.389229 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.389231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.389466 (+ 235us) server_negotiation.cc:407] Connection header received
0926 05:45:28.390003 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.390005 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.390017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.390069 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:28.390747 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.391266 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.392272 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.392485 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.392563 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.393542 (+ 979us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:28.393543 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:28.393543 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:28.393548 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:28.393562 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:28.393567 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:28.393619 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.393720 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.393778 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:45:28.394065 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.389170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:36575 (local address 127.23.213.194:57069)
0926 05:45:28.389340 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.389374 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.389895 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.390148 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.390151 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.390159 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:28.390648 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.390653 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.391372 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.391373 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.392161 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.392163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.392617 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.392619 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.392796 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.393386 (+ 590us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:28.393386 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:28.393430 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:28.393454 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:28.393628 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:28.393633 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:28.393637 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.393723 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.393849 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260926 05:45:28.394531 9785 heartbeater.cc:347] Connected to a master server at 127.23.213.254:36575
I20260926 05:45:28.394636 9785 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:28.394834 9785 heartbeater.cc:511] Master 127.23.213.254:36575 requested a full tablet report, sending...
I20260926 05:45:28.395182 9466 ts_manager.cc:194] Registered new tserver with Master: 417fd07fd2914caaae12a83053341c6b (127.23.213.194:33813)
I20260926 05:45:28.395560 9466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.194:57069
W20260926 05:45:28.474037 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:28.474233 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:28.474256 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:28.475353 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:28.475387 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:28.475412 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:45:28.476672 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:43779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:36575
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:28.476866 9788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:28.477118 9788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:28.482028 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:28.482038 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:28.482023 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:28.482352 9788 server_base.cc:1034] running on GCE node
I20260926 05:45:28.482633 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:28.482939 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:28.484082 9788 hybrid_clock.cc:648] HybridClock initialized: now 1790401528484065 us; error 31 us; skew 500 ppm
I20260926 05:45:28.485719 9788 webserver.cc:460] Webserver started at http://127.23.213.195:35455/ using document root <none> and password file <none>
I20260926 05:45:28.486037 9788 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:28.486084 9788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:28.486222 9788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:28.487299 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "78a2e0b553f6498993ef2f16a18af747"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "8fb92f7c39566810de4eb271ded141d5"
server_key_iv: "4fca3538bdb5828c2e4c3a1f9551d236"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.487664 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "78a2e0b553f6498993ef2f16a18af747"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "8fb92f7c39566810de4eb271ded141d5"
server_key_iv: "4fca3538bdb5828c2e4c3a1f9551d236"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.488961 9788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:28.489645 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.489838 9788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:28.489933 9788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "78a2e0b553f6498993ef2f16a18af747"
format_stamp: "Formatted at 2026-09-26 05:45:28 on dist-test-slave-1fdt"
server_key: "8fb92f7c39566810de4eb271ded141d5"
server_key_iv: "4fca3538bdb5828c2e4c3a1f9551d236"
server_key_version: "encryptionkey@0"
I20260926 05:45:28.490021 9788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:28.500059 9788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:28.500351 9788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:28.500502 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:28.500716 9788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:28.501051 9788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:28.501119 9788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.501159 9788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:28.501186 9788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:28.506893 9788 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:33187
I20260926 05:45:28.506935 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:33187 every 8 connection(s)
I20260926 05:45:28.507244 9788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:45:28.508721 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 9788
I20260926 05:45:28.508838 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:45:28.509022 24407 external_mini_cluster.cc:1428] Setting key a5930556137c423af464985bf4fb6bff
I20260926 05:45:28.512428 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.507928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:37623 (local address 127.23.213.254:36575)
0926 05:45:28.507990 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.507992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.508193 (+ 201us) server_negotiation.cc:407] Connection header received
0926 05:45:28.508691 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.508693 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.508707 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.508787 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:28.509665 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.510303 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.511242 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.511427 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.511477 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.512216 (+ 739us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:28.512217 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:28.512217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:28.512221 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:28.512233 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:28.512237 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:28.512294 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.512349 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.512371 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:45:28.512636 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.507882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:36575 (local address 127.23.213.195:37623)
0926 05:45:28.508064 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.508103 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.508594 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.508895 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.508899 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.508910 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:28.509568 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.509575 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.510381 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.510382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.511168 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.511170 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.511498 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.511499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.511603 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.512091 (+ 488us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:28.512091 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:28.512119 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:28.512143 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:28.512295 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:28.512299 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:28.512301 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.512351 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.512428 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260926 05:45:28.513140 9916 heartbeater.cc:347] Connected to a master server at 127.23.213.254:36575
I20260926 05:45:28.513242 9916 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:28.513415 9916 heartbeater.cc:511] Master 127.23.213.254:36575 requested a full tablet report, sending...
I20260926 05:45:28.513839 9466 ts_manager.cc:194] Registered new tserver with Master: 78a2e0b553f6498993ef2f16a18af747 (127.23.213.195:33187)
I20260926 05:45:28.514212 9466 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.195:37623
I20260926 05:45:28.520905 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:45:28.525460 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.522355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.23.213.254:36575)
0926 05:45:28.522437 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.522440 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.522530 (+ 90us) server_negotiation.cc:407] Connection header received
0926 05:45:28.522603 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.522604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.522617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.522674 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:28.522878 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.523374 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.524157 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.524359 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.524437 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.525175 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:28.525177 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:28.525177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:28.525183 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:28.525198 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:28.525202 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:28.525282 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.525346 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.525370 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 05:45:28.527208 9466 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:45:28.528734 9466 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:45:28.534917 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.531178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49444 (local address 127.23.213.194:33813)
0926 05:45:28.531332 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.531336 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.531452 (+ 116us) server_negotiation.cc:407] Connection header received
0926 05:45:28.531553 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.531555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.531572 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.531664 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:28.531843 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.532549 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.533921 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.534520 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.534613 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.534704 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.534795 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.534832 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":89,"threads_started":1}
I20260926 05:45:28.534914 9927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.531081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:33813 (local address 127.0.0.1:49444)
0926 05:45:28.531399 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.531410 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.531462 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.531732 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.531733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.531735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:28.531767 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.531768 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.532639 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.532640 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.533832 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.533834 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.534613 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.534614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.534688 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.534701 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.534782 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.534831 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":70,"threads_started":1}
I20260926 05:45:28.535239 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.530881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:33187 (local address 127.0.0.1:47950)
0926 05:45:28.531045 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.531086 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.531151 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.531554 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.531557 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.531560 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:28.531647 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.531649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.533094 (+ 1445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.533095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.534078 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.534080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.534994 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.534995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.535081 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.535085 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.535114 (+ 29us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.535152 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":1,"thread_start_us":60,"threads_started":1}
I20260926 05:45:28.535413 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.530908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:35955 (local address 127.0.0.1:38316)
0926 05:45:28.531182 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.531193 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.531244 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.531767 (+ 523us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.531768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.531769 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:28.531810 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.531812 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.533062 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.533063 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.534102 (+ 1039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.534104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.535126 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.535127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.535176 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.535179 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.535221 (+ 42us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.535258 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":85,"threads_started":1}
I20260926 05:45:28.535697 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.531219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47950 (local address 127.23.213.195:33187)
0926 05:45:28.531379 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.531383 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.531395 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:45:28.531437 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.531439 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.531455 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.531539 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:28.531911 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.532990 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.534168 (+ 1178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.534889 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.535359 (+ 470us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.535533 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.535561 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.535605 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":94,"threads_started":1}
I20260926 05:45:28.535761 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.530992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38316 (local address 127.23.213.193:35955)
0926 05:45:28.531590 (+ 598us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.531595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.531609 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:45:28.531647 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.531649 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.531663 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.531766 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:28.531868 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.532966 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.534216 (+ 1250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.534897 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.535519 (+ 622us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.535612 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.535634 (+ 22us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.535672 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":563,"thread_start_us":94,"threads_started":1}
I20260926 05:45:28.535635 9719 tablet_service.cc:1467] Processing CreateTablet for tablet 7f941ee631ba49fe8f221dfbc2a783fa (DEFAULT_TABLE table=test-table [id=df71b08a948a445e9fe58a1721551fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:28.536051 9719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f941ee631ba49fe8f221dfbc2a783fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:28.536309 9850 tablet_service.cc:1467] Processing CreateTablet for tablet 7f941ee631ba49fe8f221dfbc2a783fa (DEFAULT_TABLE table=test-table [id=df71b08a948a445e9fe58a1721551fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:28.536603 9850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f941ee631ba49fe8f221dfbc2a783fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:28.538367 9588 tablet_service.cc:1467] Processing CreateTablet for tablet 7f941ee631ba49fe8f221dfbc2a783fa (DEFAULT_TABLE table=test-table [id=df71b08a948a445e9fe58a1721551fb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:28.538712 9588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f941ee631ba49fe8f221dfbc2a783fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:28.539086 9930 tablet_bootstrap.cc:492] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Bootstrap starting.
I20260926 05:45:28.539362 9931 tablet_bootstrap.cc:492] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Bootstrap starting.
I20260926 05:45:28.539719 9930 tablet_bootstrap.cc:654] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:28.540014 9931 tablet_bootstrap.cc:654] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:28.540020 9930 log.cc:824] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:28.540323 9931 log.cc:824] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:28.540804 9930 tablet_bootstrap.cc:492] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: No bootstrap required, opened a new log
I20260926 05:45:28.541004 9930 ts_tablet_manager.cc:1397] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:45:28.541097 9931 tablet_bootstrap.cc:492] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: No bootstrap required, opened a new log
I20260926 05:45:28.541165 9931 ts_tablet_manager.cc:1397] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:28.542179 9934 tablet_bootstrap.cc:492] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828: Bootstrap starting.
I20260926 05:45:28.542886 9934 tablet_bootstrap.cc:654] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:28.543196 9934 log.cc:824] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:28.543536 9931 raft_consensus.cc:348] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.543728 9931 raft_consensus.cc:374] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:28.543784 9931 raft_consensus.cc:729] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 78a2e0b553f6498993ef2f16a18af747, State: Initialized, Role: FOLLOWER
I20260926 05:45:28.543690 9930 raft_consensus.cc:348] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.543839 9930 raft_consensus.cc:374] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:28.543881 9930 raft_consensus.cc:729] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 417fd07fd2914caaae12a83053341c6b, State: Initialized, Role: FOLLOWER
I20260926 05:45:28.543924 9931 consensus_queue.cc:260] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.543972 9934 tablet_bootstrap.cc:492] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828: No bootstrap required, opened a new log
I20260926 05:45:28.544024 9934 ts_tablet_manager.cc:1397] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:28.543999 9930 consensus_queue.cc:260] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.544253 9930 ts_tablet_manager.cc:1428] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:45:28.544390 9785 heartbeater.cc:503] Master 127.23.213.254:36575 was elected leader, sending a full tablet report...
I20260926 05:45:28.544482 9931 ts_tablet_manager.cc:1428] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:45:28.544664 9916 heartbeater.cc:503] Master 127.23.213.254:36575 was elected leader, sending a full tablet report...
I20260926 05:45:28.546284 9934 raft_consensus.cc:348] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.546396 9934 raft_consensus.cc:374] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:28.546416 9934 raft_consensus.cc:729] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7764cc8e9e8444049b57f6ed50dd1828, State: Initialized, Role: FOLLOWER
I20260926 05:45:28.546514 9934 consensus_queue.cc:260] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.546746 9934 ts_tablet_manager.cc:1428] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:45:28.546813 9654 heartbeater.cc:503] Master 127.23.213.254:36575 was elected leader, sending a full tablet report...
W20260926 05:45:28.639002 9786 tablet.cc:2369] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:28.655055 9938 raft_consensus.cc:482] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:28.655211 9938 raft_consensus.cc:504] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.655573 9938 leader_election.cc:288] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 78a2e0b553f6498993ef2f16a18af747 (127.23.213.195:33187), 417fd07fd2914caaae12a83053341c6b (127.23.213.194:33813)
I20260926 05:45:28.658500 9656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.655774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:33187 (local address 127.23.213.193:41935)
0926 05:45:28.655843 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.655855 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.655914 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.656095 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.656096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.656097 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:28.656189 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.656191 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.656882 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.656883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.657658 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.657660 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.658230 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.658231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.658312 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.658315 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.658372 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.658404 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 05:45:28.658505 9929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.655898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:41935 (local address 127.23.213.195:33187)
0926 05:45:28.655948 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.655950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.655961 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:45:28.655994 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.655995 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.656004 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.656071 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:28.656276 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.656802 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.657774 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.658149 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.658206 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.658212 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.658387 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.658434 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 05:45:28.658584 9928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.655971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:44333 (local address 127.23.213.194:33813)
0926 05:45:28.656020 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.656022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.656140 (+ 118us) server_negotiation.cc:407] Connection header received
0926 05:45:28.656188 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.656189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.656198 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.656253 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:28.656410 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.656882 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.657840 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.658211 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.658282 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.658289 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.658475 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.658515 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 05:45:28.658634 9939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.655889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:33813 (local address 127.23.213.193:44333)
0926 05:45:28.656066 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.656076 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:28.656131 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:28.656276 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:28.656277 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:28.656279 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:28.656344 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.656345 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.656965 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.656966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.657719 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:28.657721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.658299 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.658300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:28.658392 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.658396 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:28.658451 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:45:28.658485 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":104,"threads_started":1}
I20260926 05:45:28.658897 9870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f941ee631ba49fe8f221dfbc2a783fa" candidate_uuid: "7764cc8e9e8444049b57f6ed50dd1828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a2e0b553f6498993ef2f16a18af747" is_pre_election: true
I20260926 05:45:28.658941 9739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f941ee631ba49fe8f221dfbc2a783fa" candidate_uuid: "7764cc8e9e8444049b57f6ed50dd1828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417fd07fd2914caaae12a83053341c6b" is_pre_election: true
I20260926 05:45:28.659061 9870 raft_consensus.cc:2454] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7764cc8e9e8444049b57f6ed50dd1828 in term 0.
I20260926 05:45:28.659117 9739 raft_consensus.cc:2454] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7764cc8e9e8444049b57f6ed50dd1828 in term 0.
I20260926 05:45:28.659298 9542 leader_election.cc:302] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7764cc8e9e8444049b57f6ed50dd1828, 78a2e0b553f6498993ef2f16a18af747; no voters:
I20260926 05:45:28.659417 9938 raft_consensus.cc:2781] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:45:28.659469 9938 raft_consensus.cc:482] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:45:28.659485 9938 raft_consensus.cc:3037] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:28.660210 9938 raft_consensus.cc:504] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.660337 9938 leader_election.cc:288] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [CANDIDATE]: Term 1 election: Requested vote from peers 78a2e0b553f6498993ef2f16a18af747 (127.23.213.195:33187), 417fd07fd2914caaae12a83053341c6b (127.23.213.194:33813)
I20260926 05:45:28.660496 9870 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f941ee631ba49fe8f221dfbc2a783fa" candidate_uuid: "7764cc8e9e8444049b57f6ed50dd1828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "78a2e0b553f6498993ef2f16a18af747"
I20260926 05:45:28.660538 9739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f941ee631ba49fe8f221dfbc2a783fa" candidate_uuid: "7764cc8e9e8444049b57f6ed50dd1828" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "417fd07fd2914caaae12a83053341c6b"
I20260926 05:45:28.660609 9870 raft_consensus.cc:3037] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:28.660611 9739 raft_consensus.cc:3037] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:28.661288 9739 raft_consensus.cc:2454] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7764cc8e9e8444049b57f6ed50dd1828 in term 1.
I20260926 05:45:28.661437 9870 raft_consensus.cc:2454] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7764cc8e9e8444049b57f6ed50dd1828 in term 1.
I20260926 05:45:28.661454 9542 leader_election.cc:302] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 417fd07fd2914caaae12a83053341c6b, 7764cc8e9e8444049b57f6ed50dd1828; no voters:
I20260926 05:45:28.661650 9938 raft_consensus.cc:2781] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:28.661846 9938 raft_consensus.cc:686] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [term 1 LEADER]: Becoming Leader. State: Replica: 7764cc8e9e8444049b57f6ed50dd1828, State: Running, Role: LEADER
I20260926 05:45:28.661952 9938 consensus_queue.cc:237] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } }
I20260926 05:45:28.662771 9465 catalog_manager.cc:5495] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7764cc8e9e8444049b57f6ed50dd1828 (127.23.213.193). New cstate: current_term: 1 leader_uuid: "7764cc8e9e8444049b57f6ed50dd1828" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7764cc8e9e8444049b57f6ed50dd1828" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 35955 } health_report { overall_health: HEALTHY } } }
I20260926 05:45:28.680706 9926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.678582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38330 (local address 127.23.213.193:35955)
0926 05:45:28.678648 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:28.678650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:28.678662 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:45:28.678699 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:28.678700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:28.678717 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:28.678787 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:45:28.678907 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.679358 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.680054 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:28.680250 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:28.680301 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:28.680431 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:45:28.680522 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:45:28.680580 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:28.680617 (+ 37us) server_negotiation.cc:299] Negotiation successful
0926 05:45:28.680633 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:45:28.682477 9739 raft_consensus.cc:1261] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 1 FOLLOWER]: Refusing update from remote peer 7764cc8e9e8444049b57f6ed50dd1828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:28.682482 9870 raft_consensus.cc:1261] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 1 FOLLOWER]: Refusing update from remote peer 7764cc8e9e8444049b57f6ed50dd1828: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:28.682869 9940 consensus_queue.cc:1035] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "78a2e0b553f6498993ef2f16a18af747" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 33187 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:28.683043 9938 consensus_queue.cc:1035] T 7f941ee631ba49fe8f221dfbc2a783fa P 7764cc8e9e8444049b57f6ed50dd1828 [LEADER]: Connected to new peer: Peer: permanent_uuid: "417fd07fd2914caaae12a83053341c6b" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:28.685403 9943 mvcc.cc:205] Tried to move back new op lower bound from 7333484661480751104 to 7333484661399941120. Current Snapshot: MvccSnapshot[applied={T|T < 7333484661480751104}]
I20260926 05:45:28.685679 9945 mvcc.cc:205] Tried to move back new op lower bound from 7333484661480751104 to 7333484661399941120. Current Snapshot: MvccSnapshot[applied={T|T < 7333484661480751104}]
I20260926 05:45:28.687885 9465 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:28.687979 9465 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:28.688701 9465 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 15aed23cad9d4f5e8a5dba2f319cf903: Sending DeleteTablet for 3 replicas of tablet 7f941ee631ba49fe8f221dfbc2a783fa
I20260926 05:45:28.689080 9719 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f941ee631ba49fe8f221dfbc2a783fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:28 UTC) from {username='slave'} at 127.0.0.1:49444
I20260926 05:45:28.689079 9850 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f941ee631ba49fe8f221dfbc2a783fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:28 UTC) from {username='slave'} at 127.0.0.1:47950
I20260926 05:45:28.689492 9954 tablet_replica.cc:330] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: stopping tablet replica
I20260926 05:45:28.689468 9588 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f941ee631ba49fe8f221dfbc2a783fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:28 UTC) from {username='slave'} at 127.0.0.1:38316
I20260926 05:45:28.689495 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 9526
I20260926 05:45:28.690220 9954 raft_consensus.cc:2229] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:28.690328 9954 raft_consensus.cc:2258] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:28.690493 9953 tablet_replica.cc:330] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: stopping tablet replica
I20260926 05:45:28.690591 9953 raft_consensus.cc:2229] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:28.690652 9953 raft_consensus.cc:2258] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:28.690944 9954 ts_tablet_manager.cc:1905] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:28.692507 9954 ts_tablet_manager.cc:1918] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:28.692538 9953 ts_tablet_manager.cc:1905] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:28.692580 9954 log.cc:1196] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/7f941ee631ba49fe8f221dfbc2a783fa
I20260926 05:45:28.692842 9954 ts_tablet_manager.cc:1939] T 7f941ee631ba49fe8f221dfbc2a783fa P 78a2e0b553f6498993ef2f16a18af747: Deleting consensus metadata
I20260926 05:45:28.693252 9451 catalog_manager.cc:4841] TS 78a2e0b553f6498993ef2f16a18af747 (127.23.213.195:33187): tablet 7f941ee631ba49fe8f221dfbc2a783fa (table test-table [id=df71b08a948a445e9fe58a1721551fb8]) successfully deleted
I20260926 05:45:28.694011 9953 ts_tablet_manager.cc:1918] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:28.694070 9953 log.cc:1196] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/7f941ee631ba49fe8f221dfbc2a783fa
I20260926 05:45:28.694303 9953 ts_tablet_manager.cc:1939] T 7f941ee631ba49fe8f221dfbc2a783fa P 417fd07fd2914caaae12a83053341c6b: Deleting consensus metadata
I20260926 05:45:28.694729 9451 catalog_manager.cc:4841] TS 417fd07fd2914caaae12a83053341c6b (127.23.213.194:33813): tablet 7f941ee631ba49fe8f221dfbc2a783fa (table test-table [id=df71b08a948a445e9fe58a1721551fb8]) successfully deleted
W20260926 05:45:28.695452 9450 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:35955 (error 108)
I20260926 05:45:28.695719 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 9657
I20260926 05:45:28.696414 9924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:28.695846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:35955 (local address 127.0.0.1:38340)
0926 05:45:28.696326 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:28.696374 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:35955: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":455}
W20260926 05:45:28.696560 9450 catalog_manager.cc:4584] TS 7764cc8e9e8444049b57f6ed50dd1828 (127.23.213.193:35955): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7f941ee631ba49fe8f221dfbc2a783fa: Network error: Client connection negotiation failed: client connection to 127.23.213.193:35955: connect: Connection refused (error 111)
I20260926 05:45:28.701265 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 9788
I20260926 05:45:28.705731 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 9435
2026-09-26T05:45:28Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (783 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:28 dist-test-slave-1fdt krb5kdc[9958](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:28 dist-test-slave-1fdt krb5kdc[9958](info): set up 2 sockets
Sep 26 05:45:28 dist-test-slave-1fdt krb5kdc[9958](info): commencing operation
krb5kdc: starting...
W20260926 05:45:29.749439 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.024s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:29 dist-test-slave-1fdt krb5kdc[9958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:29.869181 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:46611
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:46611
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:29.951192 9971 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:29.951428 9971 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:29.951452 9971 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:29.952859 9971 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:29.952908 9971 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:29.952927 9971 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:29.952941 9971 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:29.952953 9971 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:29.954582 9971 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:46611
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:46611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9971
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:29.954850 9971 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:29.955098 9971 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:29.960071 9979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:29.960099 9976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:29.960143 9971 server_base.cc:1034] running on GCE node
W20260926 05:45:29.960107 9977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:29.960625 9971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:29.960891 9971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:29.962028 9971 hybrid_clock.cc:648] HybridClock initialized: now 1790401529962018 us; error 30 us; skew 500 ppm
Sep 26 05:45:29 dist-test-slave-1fdt krb5kdc[9958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401529, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:29.964389 9971 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:29.964865 9971 webserver.cc:460] Webserver started at http://127.23.213.254:33251/ using document root <none> and password file <none>
I20260926 05:45:29.965102 9971 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:29.965169 9971 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:29.965301 9971 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:29.966398 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "c4fb30c43eab48d58e1807980885a0fa"
format_stamp: "Formatted at 2026-09-26 05:45:29 on dist-test-slave-1fdt"
server_key: "1a670de4a49cf1fd3ec1c23a0feb341e"
server_key_iv: "48aae9279adef22f7f332597893a74d2"
server_key_version: "encryptionkey@0"
I20260926 05:45:29.966765 9971 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "c4fb30c43eab48d58e1807980885a0fa"
format_stamp: "Formatted at 2026-09-26 05:45:29 on dist-test-slave-1fdt"
server_key: "1a670de4a49cf1fd3ec1c23a0feb341e"
server_key_iv: "48aae9279adef22f7f332597893a74d2"
server_key_version: "encryptionkey@0"
I20260926 05:45:29.968046 9971 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:29.968813 9986 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:29.969117 9971 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:29.969223 9971 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "c4fb30c43eab48d58e1807980885a0fa"
format_stamp: "Formatted at 2026-09-26 05:45:29 on dist-test-slave-1fdt"
server_key: "1a670de4a49cf1fd3ec1c23a0feb341e"
server_key_iv: "48aae9279adef22f7f332597893a74d2"
server_key_version: "encryptionkey@0"
I20260926 05:45:29.969296 9971 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:29.989795 9971 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:29.991663 9971 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:29.991846 9971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:29.996016 9971 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:46611
I20260926 05:45:29.996053 10038 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:46611 every 8 connection(s)
I20260926 05:45:29.996428 9971 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:29.997323 10039 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:29.999634 10039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa: Bootstrap starting.
I20260926 05:45:30.000180 10039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:30.000423 10039 log.cc:824] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:30.001096 10039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa: No bootstrap required, opened a new log
I20260926 05:45:30.002686 10039 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fb30c43eab48d58e1807980885a0fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46611 } }
I20260926 05:45:30.002857 10039 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:30.002882 10039 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4fb30c43eab48d58e1807980885a0fa, State: Initialized, Role: FOLLOWER
I20260926 05:45:30.002985 10039 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fb30c43eab48d58e1807980885a0fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46611 } }
I20260926 05:45:30.003074 10039 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:30.003108 10039 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:30.003162 10039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:30.003749 10039 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fb30c43eab48d58e1807980885a0fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46611 } }
I20260926 05:45:30.003865 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 9971
I20260926 05:45:30.003880 10039 leader_election.cc:302] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4fb30c43eab48d58e1807980885a0fa; no voters:
I20260926 05:45:30.003978 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:30.004051 10039 leader_election.cc:288] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:30.004104 10042 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:30.004200 24407 external_mini_cluster.cc:1428] Setting key 304d27ce8eb6dbd714ebe81025c11e34
I20260926 05:45:30.004266 10042 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [term 1 LEADER]: Becoming Leader. State: Replica: c4fb30c43eab48d58e1807980885a0fa, State: Running, Role: LEADER
I20260926 05:45:30.004312 10039 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:30.004366 10042 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fb30c43eab48d58e1807980885a0fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46611 } }
I20260926 05:45:30.004851 10043 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4fb30c43eab48d58e1807980885a0fa. Latest consensus state: current_term: 1 leader_uuid: "c4fb30c43eab48d58e1807980885a0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fb30c43eab48d58e1807980885a0fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46611 } } }
I20260926 05:45:30.005092 10043 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:30.005028 10044 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4fb30c43eab48d58e1807980885a0fa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4fb30c43eab48d58e1807980885a0fa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 46611 } } }
I20260926 05:45:30.005353 10044 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:30.005673 10051 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:30.006278 10051 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:30.007895 10051 catalog_manager.cc:1347] Generated new cluster ID: ff3592fc423e4a9db306fc371594625d
I20260926 05:45:30.007951 10051 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:45:30 dist-test-slave-1fdt krb5kdc[9958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401529, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:30.015563 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.004687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41938 (local address 127.23.213.254:46611)
0926 05:45:30.004843 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.004855 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.004888 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:45:30.005009 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.005017 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.005078 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.005237 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:30.006289 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.007624 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.008528 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.009044 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.009232 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.011595 (+ 2363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:30.011598 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:30.011599 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:30.011611 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:30.014039 (+ 2428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.014409 (+ 370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.014413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.014413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.014513 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.014732 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.014733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.014733 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.014966 (+ 233us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:30.015126 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.015252 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.015393 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":62,"threads_started":1}
I20260926 05:45:30.022712 10051 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:30.023224 10051 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:30.027243 10051 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa: Generated new TSK 0
I20260926 05:45:30.027420 10051 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:45:30.061606 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46611
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:30.151082 10065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:30.151297 10065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:30.151322 10065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:30.152577 10065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:30.152627 10065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:30.152657 10065 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:45:30.154219 10065 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:30.154484 10065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:30.154736 10065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:30.160102 10073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:30.160145 10071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:30.160177 10070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:30.160835 10065 server_base.cc:1034] running on GCE node
I20260926 05:45:30.161083 10065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:30.161387 10065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:30.162549 10065 hybrid_clock.cc:648] HybridClock initialized: now 1790401530162528 us; error 50 us; skew 500 ppm
Sep 26 05:45:30 dist-test-slave-1fdt krb5kdc[9958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:30.165962 10065 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:45:30.166481 10065 webserver.cc:460] Webserver started at http://127.23.213.193:46617/ using document root <none> and password file <none>
I20260926 05:45:30.166749 10065 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:30.166843 10065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:30.166972 10065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:30.168280 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "6d07d70d61554b8eb5faa02d5f2f8d82"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "1141611cd69bd6776c858b966f6eee7d"
server_key_iv: "379a1d3e15ebdd5557f26208a3c1a147"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.168651 10065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "6d07d70d61554b8eb5faa02d5f2f8d82"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "1141611cd69bd6776c858b966f6eee7d"
server_key_iv: "379a1d3e15ebdd5557f26208a3c1a147"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.170089 10065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:45:30.170831 10080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.171000 10065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:30.171099 10065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "6d07d70d61554b8eb5faa02d5f2f8d82"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "1141611cd69bd6776c858b966f6eee7d"
server_key_iv: "379a1d3e15ebdd5557f26208a3c1a147"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.171183 10065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:30.182526 10065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:30.184299 10065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:30.184455 10065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:30.184674 10065 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:30.184994 10065 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:30.185055 10065 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.185096 10065 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:30.185124 10065 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.190871 10065 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:41399
I20260926 05:45:30.190958 10193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:41399 every 8 connection(s)
I20260926 05:45:30.191293 10065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:45:30.197103 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 10065
I20260926 05:45:30.197198 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:45:30.197405 24407 external_mini_cluster.cc:1428] Setting key 3b6b4b36fcb1fc5d46afa1bc4544c457
Sep 26 05:45:30 dist-test-slave-1fdt krb5kdc[9958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:30.201112 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.192181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:60519 (local address 127.23.213.254:46611)
0926 05:45:30.192261 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.192263 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.192437 (+ 174us) server_negotiation.cc:407] Connection header received
0926 05:45:30.192951 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.192952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.192972 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.193041 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:30.194009 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.194606 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.195550 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.195770 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.195828 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.198105 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:30.198107 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:30.198108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:30.198115 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:30.199790 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.200175 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.200176 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.200177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.200243 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.200470 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.200471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.200471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.200602 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:30.200657 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.200906 (+ 249us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.201008 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 05:45:30.201128 10196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.192159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46611 (local address 127.23.213.193:60519)
0926 05:45:30.192323 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.192364 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.192855 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.193094 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.193097 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.193646 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:30.193917 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.193922 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.194690 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.194690 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.195469 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.195471 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.195847 (+ 376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.195848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.195942 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.196373 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:30.196380 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:30.197999 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:30.199887 (+ 1888us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:30.199893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:30.199895 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:30.200103 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:30.200312 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:30.200313 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:30.200313 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:30.200404 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:30.200660 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:30.200664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:30.200779 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.200863 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.200976 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":78,"threads_started":1}
I20260926 05:45:30.201690 10194 heartbeater.cc:347] Connected to a master server at 127.23.213.254:46611
I20260926 05:45:30.201830 10194 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:30.202029 10194 heartbeater.cc:511] Master 127.23.213.254:46611 requested a full tablet report, sending...
I20260926 05:45:30.202633 10003 ts_manager.cc:194] Registered new tserver with Master: 6d07d70d61554b8eb5faa02d5f2f8d82 (127.23.213.193:41399)
I20260926 05:45:30.203646 10003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:60519
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:45:30.224864 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46611
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:30.306743 10199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:30.306933 10199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:30.306958 10199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:30.308015 10199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:30.308048 10199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:30.308074 10199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:45:30.309398 10199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:30.309566 10199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:30.309723 10199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:30.314603 10205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:30.314568 10207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:30.314569 10204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:30.314894 10199 server_base.cc:1034] running on GCE node
I20260926 05:45:30.315101 10199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:30.315369 10199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:30.316530 10199 hybrid_clock.cc:648] HybridClock initialized: now 1790401530316527 us; error 34 us; skew 500 ppm
Sep 26 05:45:30 dist-test-slave-1fdt krb5kdc[9958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:30.319053 10199 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:45:30.319504 10199 webserver.cc:460] Webserver started at http://127.23.213.194:39319/ using document root <none> and password file <none>
I20260926 05:45:30.319725 10199 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:30.319794 10199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:30.319944 10199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:30.320978 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "2189aad7660742bba215286c5bc5c67c"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "bdc9f7087488ce5884483aef69945ddc"
server_key_iv: "89c4a73ef1d9f3dbae5e9dff1f44b9c6"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.321337 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "2189aad7660742bba215286c5bc5c67c"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "bdc9f7087488ce5884483aef69945ddc"
server_key_iv: "89c4a73ef1d9f3dbae5e9dff1f44b9c6"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.322744 10199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:30.323464 10214 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.323638 10199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:45:30.323745 10199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "2189aad7660742bba215286c5bc5c67c"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "bdc9f7087488ce5884483aef69945ddc"
server_key_iv: "89c4a73ef1d9f3dbae5e9dff1f44b9c6"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.323832 10199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:30.338395 10199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:30.340747 10199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:30.340921 10199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:30.341157 10199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:30.341461 10199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:30.341514 10199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.341550 10199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:30.341578 10199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.347240 10199 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:35805
I20260926 05:45:30.347323 10327 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:35805 every 8 connection(s)
I20260926 05:45:30.347642 10199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:45:30.350001 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 10199
I20260926 05:45:30.350118 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:45:30.350337 24407 external_mini_cluster.cc:1428] Setting key 97e3dd225ea2e472ae6210c543be77f6
Sep 26 05:45:30 dist-test-slave-1fdt krb5kdc[9958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:30.357836 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.348481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:40375 (local address 127.23.213.254:46611)
0926 05:45:30.348572 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.348574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.348729 (+ 155us) server_negotiation.cc:407] Connection header received
0926 05:45:30.349191 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.349192 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.349202 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.349254 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:30.350345 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.350953 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.351955 (+ 1002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.352272 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.352343 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.354841 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:30.354843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:30.354843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:30.354849 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:30.356326 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.356810 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.356811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.356811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.356860 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.357141 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.357142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.357142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.357241 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:30.357288 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.357618 (+ 330us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.357692 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 05:45:30.358143 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.348426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46611 (local address 127.23.213.194:40375)
0926 05:45:30.348602 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.348638 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.349105 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.349314 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.349317 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.349956 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:30.350242 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.350248 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.351066 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.351067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.351859 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.351861 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.352352 (+ 491us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.352354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.352486 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.353003 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:30.353010 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:30.354738 (+ 1728us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:30.356410 (+ 1672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:30.356418 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:30.356420 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:30.356730 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:30.356929 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:30.356930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:30.356930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:30.357073 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:30.357307 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:30.357313 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:30.357477 (+ 164us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.357602 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.357809 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":98,"threads_started":1}
I20260926 05:45:30.358672 10328 heartbeater.cc:347] Connected to a master server at 127.23.213.254:46611
I20260926 05:45:30.358796 10328 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
I20260926 05:45:30.358984 10328 heartbeater.cc:511] Master 127.23.213.254:46611 requested a full tablet report, sending...
I20260926 05:45:30.359400 10003 ts_manager.cc:194] Registered new tserver with Master: 2189aad7660742bba215286c5bc5c67c (127.23.213.194:35805)
I20260926 05:45:30.359800 10003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:40375
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:45:30.376309 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46611
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:30.456748 10333 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:30.456987 10333 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:30.457023 10333 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:30.458189 10333 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:30.458267 10333 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:30.458314 10333 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:45:30.459786 10333 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:46611
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10333
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:30.460047 10333 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:30.460296 10333 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:30.465240 10338 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:30.465286 10339 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:30.465438 10333 server_base.cc:1034] running on GCE node
W20260926 05:45:30.465246 10341 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:30.465793 10333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:30.466068 10333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:30.467229 10333 hybrid_clock.cc:648] HybridClock initialized: now 1790401530467217 us; error 23 us; skew 500 ppm
Sep 26 05:45:30 dist-test-slave-1fdt krb5kdc[9958](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:30.469702 10333 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:45:30.470144 10333 webserver.cc:460] Webserver started at http://127.23.213.195:45225/ using document root <none> and password file <none>
I20260926 05:45:30.470367 10333 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:30.470434 10333 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:30.470607 10333 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:30.471644 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "104f3671465642a9a3f1d8833b26747a"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "36811d35203e2b7661d8d70d8119f772"
server_key_iv: "0f53d105e2c977243034cc07358b7378"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.472025 10333 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "104f3671465642a9a3f1d8833b26747a"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "36811d35203e2b7661d8d70d8119f772"
server_key_iv: "0f53d105e2c977243034cc07358b7378"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.473348 10333 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 05:45:30.474148 10348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.474320 10333 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.474417 10333 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "104f3671465642a9a3f1d8833b26747a"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "36811d35203e2b7661d8d70d8119f772"
server_key_iv: "0f53d105e2c977243034cc07358b7378"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.474498 10333 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:30.498005 10333 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:30.500093 10333 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:30.500227 10333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:30.500398 10333 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:30.500721 10333 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:30.500756 10333 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.500780 10333 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:30.500829 10333 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.506687 10333 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:36575
I20260926 05:45:30.506767 10461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:36575 every 8 connection(s)
I20260926 05:45:30.507082 10333 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:45:30.511219 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 10333
I20260926 05:45:30.511304 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:45:30.511502 24407 external_mini_cluster.cc:1428] Setting key 1cab371f0a14015c4bf2fd27ab33dd58
Sep 26 05:45:30 dist-test-slave-1fdt krb5kdc[9958](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401530, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:30.516637 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.507773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:36445 (local address 127.23.213.254:46611)
0926 05:45:30.507839 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.507841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.508061 (+ 220us) server_negotiation.cc:407] Connection header received
0926 05:45:30.508594 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.508595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.508607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.508665 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:30.509610 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.510246 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.511384 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.511593 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.511679 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.514039 (+ 2360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:30.514041 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:30.514041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:30.514048 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:30.515505 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.515866 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.515867 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.515867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.515920 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.516126 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.516127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.516127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.516232 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:30.516282 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.516514 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.516575 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 05:45:30.516789 10464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.507778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:46611 (local address 127.23.213.195:36445)
0926 05:45:30.507946 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.507981 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.508482 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.508736 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.508739 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.509310 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:30.509514 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.509518 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.510337 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.510339 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.511302 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.511305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.511696 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.511697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.511883 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.512540 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:30.512548 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:30.513947 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:30.515586 (+ 1639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:30.515592 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:30.515593 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:30.515800 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:30.515978 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:30.515979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:30.515979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:30.516072 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:30.516292 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:30.516296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:30.516405 (+ 109us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.516484 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.516598 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":78,"threads_started":1}
I20260926 05:45:30.517244 10462 heartbeater.cc:347] Connected to a master server at 127.23.213.254:46611
I20260926 05:45:30.517339 10462 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:30.517474 10462 heartbeater.cc:511] Master 127.23.213.254:46611 requested a full tablet report, sending...
I20260926 05:45:30.517819 10003 ts_manager.cc:194] Registered new tserver with Master: 104f3671465642a9a3f1d8833b26747a (127.23.213.195:36575)
I20260926 05:45:30.518199 10003 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:36445
I20260926 05:45:30.523257 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:45:30.530850 10047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.524636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41948 (local address 127.23.213.254:46611)
0926 05:45:30.524699 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.524701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.524820 (+ 119us) server_negotiation.cc:407] Connection header received
0926 05:45:30.524857 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.524858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.524866 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.524910 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:30.525402 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.525931 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.526713 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.526900 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.526968 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.528329 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:30.528331 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:30.528331 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:30.528337 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:30.529831 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.530210 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.530211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.530211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.530261 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:30.530443 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:30.530444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:30.530444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:30.530566 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:30.530640 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.530740 (+ 100us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.530789 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:45:30.532497 10003 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:45:30.534073 10003 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:45:30.540624 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.536478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39234 (local address 127.23.213.194:35805)
0926 05:45:30.536656 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.536662 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.536671 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:45:30.536711 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.536714 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.536728 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.536832 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:30.537008 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.538141 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.539596 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.540254 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.540319 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.540364 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.540500 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.540543 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":69,"threads_started":1}
I20260926 05:45:30.540647 10473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.536279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:35805 (local address 127.0.0.1:39234)
0926 05:45:30.536426 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.536464 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.536523 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.536836 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.536839 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.536841 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:30.536922 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.536924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.538239 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.538240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.539518 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.539519 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.540323 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.540323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.540384 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.540398 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.540475 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.540543 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":60,"thread_start_us":79,"threads_started":1}
I20260926 05:45:30.540645 10476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.536279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:36575 (local address 127.0.0.1:45400)
0926 05:45:30.536569 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.536580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.536638 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.537187 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.537188 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.537191 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:30.537227 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.537228 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.538452 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.538453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.539377 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.539380 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.540302 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.540303 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.540384 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.540399 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.540475 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.540543 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":268,"mutex_wait_us":4,"thread_start_us":44,"threads_started":1}
I20260926 05:45:30.540876 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.536368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45400 (local address 127.23.213.195:36575)
0926 05:45:30.536969 (+ 601us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.536973 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.536983 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:45:30.537019 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.537021 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.537038 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.537111 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:30.537319 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.538346 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.539461 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.540191 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.540286 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.540351 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.540748 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.540803 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":76,"threads_started":1}
I20260926 05:45:30.541518 10262 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 (DEFAULT_TABLE table=test-table [id=271aea0cf9614768a44ddf811f189b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:30.541630 10396 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 (DEFAULT_TABLE table=test-table [id=271aea0cf9614768a44ddf811f189b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:30.541916 10262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:30.542035 10396 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:30.542219 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.536451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41399 (local address 127.0.0.1:54430)
0926 05:45:30.536629 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.536638 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.536696 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.537030 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.537031 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.537033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:30.537081 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.537083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.538106 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.538107 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.539510 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.539513 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.541944 (+ 2431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.541945 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.542021 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.542025 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.542097 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.542143 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160,"mutex_wait_us":35,"thread_start_us":74,"threads_started":1}
I20260926 05:45:30.542229 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.536678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54430 (local address 127.23.213.193:41399)
0926 05:45:30.536829 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.536833 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.536842 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:45:30.536886 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.536887 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.536899 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.537165 (+ 266us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:30.537249 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.538012 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.540407 (+ 2395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.541834 (+ 1427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.541942 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.542000 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.542113 (+ 113us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.542152 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260926 05:45:30.544621 10479 tablet_bootstrap.cc:492] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Bootstrap starting.
I20260926 05:45:30.544878 10480 tablet_bootstrap.cc:492] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Bootstrap starting.
I20260926 05:45:30.545159 10128 tablet_service.cc:1467] Processing CreateTablet for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 (DEFAULT_TABLE table=test-table [id=271aea0cf9614768a44ddf811f189b4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:30.545339 10479 tablet_bootstrap.cc:654] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:30.545445 10480 tablet_bootstrap.cc:654] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:30.545506 10128 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:30.545915 10479 log.cc:824] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:30.546049 10480 log.cc:824] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:30.546628 10479 tablet_bootstrap.cc:492] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: No bootstrap required, opened a new log
I20260926 05:45:30.546708 10479 ts_tablet_manager.cc:1397] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:45:30.548211 10479 raft_consensus.cc:348] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.548377 10479 raft_consensus.cc:374] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:30.548419 10479 raft_consensus.cc:729] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2189aad7660742bba215286c5bc5c67c, State: Initialized, Role: FOLLOWER
I20260926 05:45:30.548406 10483 tablet_bootstrap.cc:492] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: Bootstrap starting.
I20260926 05:45:30.548530 10479 consensus_queue.cc:260] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.548818 10479 ts_tablet_manager.cc:1428] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:45:30.548837 10328 heartbeater.cc:503] Master 127.23.213.254:46611 was elected leader, sending a full tablet report...
I20260926 05:45:30.549069 10483 tablet_bootstrap.cc:654] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:30.549211 10480 tablet_bootstrap.cc:492] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: No bootstrap required, opened a new log
I20260926 05:45:30.549288 10480 ts_tablet_manager.cc:1397] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 05:45:30.549646 10483 log.cc:824] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:30.550406 10483 tablet_bootstrap.cc:492] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: No bootstrap required, opened a new log
I20260926 05:45:30.550483 10483 ts_tablet_manager.cc:1397] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:45:30.550961 10480 raft_consensus.cc:348] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.551085 10480 raft_consensus.cc:374] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:30.551106 10480 raft_consensus.cc:729] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 104f3671465642a9a3f1d8833b26747a, State: Initialized, Role: FOLLOWER
I20260926 05:45:30.551218 10480 consensus_queue.cc:260] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.551491 10462 heartbeater.cc:503] Master 127.23.213.254:46611 was elected leader, sending a full tablet report...
I20260926 05:45:30.551488 10480 ts_tablet_manager.cc:1428] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:45:30.552186 10483 raft_consensus.cc:348] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.552357 10483 raft_consensus.cc:374] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:30.552393 10483 raft_consensus.cc:729] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6d07d70d61554b8eb5faa02d5f2f8d82, State: Initialized, Role: FOLLOWER
I20260926 05:45:30.552490 10483 consensus_queue.cc:260] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.552739 10483 ts_tablet_manager.cc:1428] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:45:30.552796 10194 heartbeater.cc:503] Master 127.23.213.254:46611 was elected leader, sending a full tablet report...
W20260926 05:45:30.598351 10329 tablet.cc:2369] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:45:30.693167 10195 tablet.cc:2369] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:30.696143 10484 raft_consensus.cc:482] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:30.696259 10484 raft_consensus.cc:504] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.696625 10484 leader_election.cc:288] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6d07d70d61554b8eb5faa02d5f2f8d82 (127.23.213.193:41399), 104f3671465642a9a3f1d8833b26747a (127.23.213.195:36575)
I20260926 05:45:30.699402 10330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.696796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41399 (local address 127.23.213.194:32829)
0926 05:45:30.696854 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.696867 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.696919 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.697077 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.697077 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.697079 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:30.697163 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.697164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.697838 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.697839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.698565 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.698566 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.699125 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.699125 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.699215 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.699218 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.699280 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.699326 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:45:30.699402 10478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.696847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:32829 (local address 127.23.213.193:41399)
0926 05:45:30.696893 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.696896 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.696908 (+ 12us) server_negotiation.cc:407] Connection header received
0926 05:45:30.696988 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.696989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.696998 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.697073 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:30.697236 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.697716 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.698657 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.699037 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.699118 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.699127 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.699302 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.699339 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:45:30.699679 10475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.696898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:44725 (local address 127.23.213.195:36575)
0926 05:45:30.696955 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.696957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.697152 (+ 195us) server_negotiation.cc:407] Connection header received
0926 05:45:30.697225 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.697226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.697235 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.697314 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:30.697469 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.698004 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.698917 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.699293 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.699376 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.699383 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.699570 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.699618 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:45:30.699800 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef9a1779fef42f291ba9e7a5e6b66b8" candidate_uuid: "2189aad7660742bba215286c5bc5c67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" is_pre_election: true
I20260926 05:45:30.699896 10488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.696797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:36575 (local address 127.23.213.194:44725)
0926 05:45:30.697062 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.697072 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:30.697161 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:30.697318 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:30.697319 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:30.697320 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:30.697395 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.697396 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.698082 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.698083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.698808 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:30.698810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.699375 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.699376 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:30.699473 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.699477 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:30.699765 (+ 288us) client_negotiation.cc:240] Negotiation successful
0926 05:45:30.699815 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":79,"thread_start_us":137,"threads_started":1}
I20260926 05:45:30.699977 10148 raft_consensus.cc:2454] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2189aad7660742bba215286c5bc5c67c in term 0.
I20260926 05:45:30.700222 10218 leader_election.cc:302] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2189aad7660742bba215286c5bc5c67c, 6d07d70d61554b8eb5faa02d5f2f8d82; no voters:
I20260926 05:45:30.700225 10416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef9a1779fef42f291ba9e7a5e6b66b8" candidate_uuid: "2189aad7660742bba215286c5bc5c67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104f3671465642a9a3f1d8833b26747a" is_pre_election: true
I20260926 05:45:30.700356 10484 raft_consensus.cc:2781] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:45:30.700368 10416 raft_consensus.cc:2454] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2189aad7660742bba215286c5bc5c67c in term 0.
I20260926 05:45:30.700412 10484 raft_consensus.cc:482] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:45:30.700438 10484 raft_consensus.cc:3037] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:30.701192 10484 raft_consensus.cc:504] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.701333 10484 leader_election.cc:288] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [CANDIDATE]: Term 1 election: Requested vote from peers 6d07d70d61554b8eb5faa02d5f2f8d82 (127.23.213.193:41399), 104f3671465642a9a3f1d8833b26747a (127.23.213.195:36575)
I20260926 05:45:30.701522 10148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef9a1779fef42f291ba9e7a5e6b66b8" candidate_uuid: "2189aad7660742bba215286c5bc5c67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82"
I20260926 05:45:30.701579 10416 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ef9a1779fef42f291ba9e7a5e6b66b8" candidate_uuid: "2189aad7660742bba215286c5bc5c67c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "104f3671465642a9a3f1d8833b26747a"
I20260926 05:45:30.701607 10148 raft_consensus.cc:3037] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:30.701663 10416 raft_consensus.cc:3037] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:30.702342 10148 raft_consensus.cc:2454] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2189aad7660742bba215286c5bc5c67c in term 1.
I20260926 05:45:30.702421 10416 raft_consensus.cc:2454] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2189aad7660742bba215286c5bc5c67c in term 1.
I20260926 05:45:30.702497 10218 leader_election.cc:302] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2189aad7660742bba215286c5bc5c67c, 6d07d70d61554b8eb5faa02d5f2f8d82; no voters:
I20260926 05:45:30.702569 10484 raft_consensus.cc:2781] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:30.702713 10484 raft_consensus.cc:686] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 1 LEADER]: Becoming Leader. State: Replica: 2189aad7660742bba215286c5bc5c67c, State: Running, Role: LEADER
I20260926 05:45:30.702828 10484 consensus_queue.cc:237] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } }
I20260926 05:45:30.703604 10003 catalog_manager.cc:5495] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c reported cstate change: term changed from 0 to 1, leader changed from <none> to 2189aad7660742bba215286c5bc5c67c (127.23.213.194). New cstate: current_term: 1 leader_uuid: "2189aad7660742bba215286c5bc5c67c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2189aad7660742bba215286c5bc5c67c" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35805 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 } health_report { overall_health: UNKNOWN } } }
I20260926 05:45:30.720969 10477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.718609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39240 (local address 127.23.213.194:35805)
0926 05:45:30.718675 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.718677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.718698 (+ 21us) server_negotiation.cc:407] Connection header received
0926 05:45:30.718772 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.718772 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.718799 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.718874 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:45:30.719008 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.719466 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.720220 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.720416 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.720484 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.720642 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:45:30.720737 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:45:30.720847 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.720878 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.720886 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:45:30.722733 10148 raft_consensus.cc:1261] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 1 FOLLOWER]: Refusing update from remote peer 2189aad7660742bba215286c5bc5c67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:30.722734 10416 raft_consensus.cc:1261] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 1 FOLLOWER]: Refusing update from remote peer 2189aad7660742bba215286c5bc5c67c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:30.723119 10484 consensus_queue.cc:1035] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "104f3671465642a9a3f1d8833b26747a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 36575 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:30.723266 10489 consensus_queue.cc:1035] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [LEADER]: Connected to new peer: Peer: permanent_uuid: "6d07d70d61554b8eb5faa02d5f2f8d82" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 41399 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:30.728264 10003 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:30.728370 10003 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:30.729090 10003 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c4fb30c43eab48d58e1807980885a0fa: Sending DeleteTablet for 3 replicas of tablet 7ef9a1779fef42f291ba9e7a5e6b66b8
I20260926 05:45:30.729430 10396 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:30 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:45400
I20260926 05:45:30.729439 10128 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:30 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:54430
I20260926 05:45:30.729661 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 10065
I20260926 05:45:30.729676 10262 tablet_service.cc:1514] Processing DeleteTablet for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:30 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:39234
I20260926 05:45:30.729810 10503 tablet_replica.cc:330] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82: stopping tablet replica
I20260926 05:45:30.729898 10503 raft_consensus.cc:2229] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:30.729985 10503 raft_consensus.cc:2258] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 6d07d70d61554b8eb5faa02d5f2f8d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:30.730186 10502 tablet_replica.cc:330] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: stopping tablet replica
I20260926 05:45:30.730268 10502 raft_consensus.cc:2229] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:30.730355 10502 raft_consensus.cc:2258] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:30.730931 10504 tablet_replica.cc:330] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: stopping tablet replica
I20260926 05:45:30.731048 10504 raft_consensus.cc:2229] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:45:30.731134 10504 raft_consensus.cc:2258] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:30.731472 10504 ts_tablet_manager.cc:1905] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:30.731506 10502 ts_tablet_manager.cc:1905] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:30.733378 10502 ts_tablet_manager.cc:1918] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:30.733379 10504 ts_tablet_manager.cc:1918] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:30.733450 10504 log.cc:1196] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/7ef9a1779fef42f291ba9e7a5e6b66b8
I20260926 05:45:30.733556 10502 log.cc:1196] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/7ef9a1779fef42f291ba9e7a5e6b66b8
I20260926 05:45:30.733734 10504 ts_tablet_manager.cc:1939] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 2189aad7660742bba215286c5bc5c67c: Deleting consensus metadata
I20260926 05:45:30.733973 10502 ts_tablet_manager.cc:1939] T 7ef9a1779fef42f291ba9e7a5e6b66b8 P 104f3671465642a9a3f1d8833b26747a: Deleting consensus metadata
I20260926 05:45:30.734153 9990 catalog_manager.cc:4841] TS 2189aad7660742bba215286c5bc5c67c (127.23.213.194:35805): tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 (table test-table [id=271aea0cf9614768a44ddf811f189b4e]) successfully deleted
I20260926 05:45:30.734455 9987 catalog_manager.cc:4841] TS 104f3671465642a9a3f1d8833b26747a (127.23.213.195:36575): tablet 7ef9a1779fef42f291ba9e7a5e6b66b8 (table test-table [id=271aea0cf9614768a44ddf811f189b4e]) successfully deleted
W20260926 05:45:30.737066 9990 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:41399 (error 108)
I20260926 05:45:30.737291 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 10199
I20260926 05:45:30.737555 10474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.737435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:41399 (local address 127.0.0.1:54440)
0926 05:45:30.737478 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:30.737517 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:41399: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 05:45:30.737887 9990 catalog_manager.cc:4584] TS 6d07d70d61554b8eb5faa02d5f2f8d82 (127.23.213.193:41399): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7ef9a1779fef42f291ba9e7a5e6b66b8: Network error: Client connection negotiation failed: client connection to 127.23.213.193:41399: connect: Connection refused (error 111)
I20260926 05:45:30.742846 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 10333
I20260926 05:45:30.747530 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 9971
2026-09-26T05:45:30Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2042 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T05:45:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:30Z Disabled control of system clock
I20260926 05:45:30.774623 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40085
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:40085
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:30.854254 10509 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:30.854465 10509 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:30.854497 10509 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:30.855558 10509 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:30.855592 10509 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:30.855604 10509 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:30.855616 10509 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:30.855629 10509 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:30.856887 10509 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:40085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10509
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:30.857075 10509 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:30.857302 10509 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:30.862049 10514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:30.862042 10515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:30.862110 10509 server_base.cc:1034] running on GCE node
W20260926 05:45:30.862042 10517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:30.862519 10509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:30.862754 10509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:30.863977 10509 hybrid_clock.cc:648] HybridClock initialized: now 1790401530863964 us; error 50 us; skew 500 ppm
I20260926 05:45:30.865370 10509 webserver.cc:460] Webserver started at http://127.23.213.254:38501/ using document root <none> and password file <none>
I20260926 05:45:30.865631 10509 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:30.865705 10509 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:30.865878 10509 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:30.866935 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "411f86051ce14f758d27a4136b124688"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "be05b8adbdaf071a1809f135fea753db"
server_key_iv: "7beb3ae613dfc0ee51e27a7ede5b479c"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.867237 10509 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "411f86051ce14f758d27a4136b124688"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "be05b8adbdaf071a1809f135fea753db"
server_key_iv: "7beb3ae613dfc0ee51e27a7ede5b479c"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.868475 10509 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:30.869097 10523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:30.869269 10509 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:30.869355 10509 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "411f86051ce14f758d27a4136b124688"
format_stamp: "Formatted at 2026-09-26 05:45:30 on dist-test-slave-1fdt"
server_key: "be05b8adbdaf071a1809f135fea753db"
server_key_iv: "7beb3ae613dfc0ee51e27a7ede5b479c"
server_key_version: "encryptionkey@0"
I20260926 05:45:30.869428 10509 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:30.881501 10509 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:30.881844 10509 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:30.881999 10509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:30.885982 10509 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:40085
I20260926 05:45:30.886024 10575 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:40085 every 8 connection(s)
I20260926 05:45:30.886361 10509 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:30.887295 10576 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:30.888603 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 10509
I20260926 05:45:30.888727 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:30.888926 24407 external_mini_cluster.cc:1428] Setting key 942f928797852d303223db1fd48d79f1
I20260926 05:45:30.889856 10576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688: Bootstrap starting.
I20260926 05:45:30.890435 10576 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:30.890741 10576 log.cc:824] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:30.891435 10576 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688: No bootstrap required, opened a new log
I20260926 05:45:30.893154 10576 raft_consensus.cc:348] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411f86051ce14f758d27a4136b124688" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40085 } }
I20260926 05:45:30.893306 10576 raft_consensus.cc:374] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:30.893368 10576 raft_consensus.cc:729] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 411f86051ce14f758d27a4136b124688, State: Initialized, Role: FOLLOWER
I20260926 05:45:30.893496 10576 consensus_queue.cc:260] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411f86051ce14f758d27a4136b124688" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40085 } }
I20260926 05:45:30.893607 10576 raft_consensus.cc:388] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:30.893651 10576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:30.893708 10576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:30.894255 10576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411f86051ce14f758d27a4136b124688" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40085 } }
I20260926 05:45:30.894374 10576 leader_election.cc:302] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 411f86051ce14f758d27a4136b124688; no voters:
I20260926 05:45:30.894537 10576 leader_election.cc:288] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:30.894577 10581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:30.894738 10581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [term 1 LEADER]: Becoming Leader. State: Replica: 411f86051ce14f758d27a4136b124688, State: Running, Role: LEADER
I20260926 05:45:30.894783 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:30.889522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44266 (local address 127.23.213.254:40085)
0926 05:45:30.889735 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:30.889779 (+ 44us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:30.889811 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:45:30.890267 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:30.890274 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:30.890331 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:30.890494 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:30.890693 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.891780 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.892597 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:30.893002 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:30.893141 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:30.893896 (+ 755us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:30.893900 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:30.893900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:30.894071 (+ 171us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:30.894106 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:30.894116 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:30.894185 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:30.894262 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 05:45:30.894438 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":80,"threads_started":1}
I20260926 05:45:30.894824 10581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411f86051ce14f758d27a4136b124688" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40085 } }
I20260926 05:45:30.894861 10576 sys_catalog.cc:564] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:30.895129 10581 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 411f86051ce14f758d27a4136b124688. Latest consensus state: current_term: 1 leader_uuid: "411f86051ce14f758d27a4136b124688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411f86051ce14f758d27a4136b124688" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40085 } } }
I20260926 05:45:30.895205 10581 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:30.895136 10583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "411f86051ce14f758d27a4136b124688" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "411f86051ce14f758d27a4136b124688" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40085 } } }
I20260926 05:45:30.895385 10586 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:30.895401 10583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:30.895892 10586 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:30.897518 10586 catalog_manager.cc:1347] Generated new cluster ID: 169608ac202542a8a6c04e8fe1c26f61
I20260926 05:45:30.897584 10586 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:30.933794 10586 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:30.934537 10586 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:30.941293 10586 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688: Generated new TSK 0
I20260926 05:45:30.941476 10586 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:30.949486 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40085
--builtin_ntp_servers=127.23.213.212:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:31.030267 10600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:31.030519 10600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:31.030555 10600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:31.031695 10600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:31.031770 10600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:31.031816 10600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:45:31.033171 10600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10600
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:31.033432 10600 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:31.033728 10600 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:31.038748 10605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:31.038748 10606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:31.038779 10608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:31.039541 10600 server_base.cc:1034] running on GCE node
I20260926 05:45:31.039723 10600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:31.039918 10600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:31.041061 10600 hybrid_clock.cc:648] HybridClock initialized: now 1790401531041050 us; error 28 us; skew 500 ppm
I20260926 05:45:31.046968 10600 webserver.cc:460] Webserver started at http://127.23.213.193:33791/ using document root <none> and password file <none>
I20260926 05:45:31.047225 10600 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:31.047272 10600 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:31.047442 10600 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:31.048532 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "d3bfa171e74b401fbf62862207af1ac6"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "572985e3b3cce273963328da5fec1064"
server_key_iv: "2ce35939aa8bd17cec8cc89fa0fe85b4"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.048897 10600 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "d3bfa171e74b401fbf62862207af1ac6"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "572985e3b3cce273963328da5fec1064"
server_key_iv: "2ce35939aa8bd17cec8cc89fa0fe85b4"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.050268 10600 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:31.051113 10614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.051313 10600 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:31.051416 10600 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "d3bfa171e74b401fbf62862207af1ac6"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "572985e3b3cce273963328da5fec1064"
server_key_iv: "2ce35939aa8bd17cec8cc89fa0fe85b4"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.051492 10600 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:31.078847 10600 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:31.079228 10600 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:31.079391 10600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:31.079640 10600 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:31.079982 10600 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:31.080015 10600 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.080039 10600 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:31.080091 10600 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.086031 10600 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:37377
I20260926 05:45:31.086109 10727 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:37377 every 8 connection(s)
I20260926 05:45:31.086472 10600 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:45:31.091938 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.087228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:59397 (local address 127.23.213.254:40085)
0926 05:45:31.087295 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.087297 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.087541 (+ 244us) server_negotiation.cc:407] Connection header received
0926 05:45:31.088182 (+ 641us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.088183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.088196 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.088255 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:31.088961 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.089475 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.090566 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.090769 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.090848 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.091663 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:31.091665 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:31.091665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:31.091670 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:31.091683 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:31.091687 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:31.091739 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.091834 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.091860 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 05:45:31.092167 10730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.087214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40085 (local address 127.23.213.193:59397)
0926 05:45:31.087421 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.087455 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.088030 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.088338 (+ 308us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.088342 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.088351 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:31.088856 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.088861 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.089571 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.089572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.090458 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.090461 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.090855 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.090856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.090995 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.091494 (+ 499us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:31.091494 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:31.091522 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:31.091544 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:31.091744 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:31.091749 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:31.091752 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.091836 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.091936 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":94,"threads_started":1}
I20260926 05:45:31.092682 10728 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40085
I20260926 05:45:31.092777 10728 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:31.092931 10728 heartbeater.cc:511] Master 127.23.213.254:40085 requested a full tablet report, sending...
I20260926 05:45:31.093438 10540 ts_manager.cc:194] Registered new tserver with Master: d3bfa171e74b401fbf62862207af1ac6 (127.23.213.193:37377)
I20260926 05:45:31.094144 10540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.193:59397
I20260926 05:45:31.094898 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 10600
I20260926 05:45:31.095028 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:45:31.095268 24407 external_mini_cluster.cc:1428] Setting key 7d03afc999e6c859bc1902f075c63a4e
I20260926 05:45:31.096149 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40085
--builtin_ntp_servers=127.23.213.212:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:31.176580 10731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:31.176815 10731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:31.176852 10731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:31.178042 10731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:31.178107 10731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:31.178146 10731 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:45:31.179492 10731 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10731
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:31.179754 10731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:31.180053 10731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:31.185072 10737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:31.185079 10736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:31.185113 10739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:31.185591 10731 server_base.cc:1034] running on GCE node
I20260926 05:45:31.185876 10731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:31.186085 10731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:31.187228 10731 hybrid_clock.cc:648] HybridClock initialized: now 1790401531187219 us; error 25 us; skew 500 ppm
I20260926 05:45:31.188406 10731 webserver.cc:460] Webserver started at http://127.23.213.194:40587/ using document root <none> and password file <none>
I20260926 05:45:31.188658 10731 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:31.188741 10731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:31.188887 10731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:31.190037 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "2c1908aac48948be856e4447a9f1c741"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "948ca9247772147662d2478097b4f52c"
server_key_iv: "6ad3ff8f2356bf9d39ecfa5cbef37ac8"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.190431 10731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "2c1908aac48948be856e4447a9f1c741"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "948ca9247772147662d2478097b4f52c"
server_key_iv: "6ad3ff8f2356bf9d39ecfa5cbef37ac8"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.191704 10731 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:31.192404 10745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.192576 10731 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:31.192683 10731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "2c1908aac48948be856e4447a9f1c741"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "948ca9247772147662d2478097b4f52c"
server_key_iv: "6ad3ff8f2356bf9d39ecfa5cbef37ac8"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.192757 10731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:31.206015 10731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:31.206319 10731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:31.206470 10731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:31.206712 10731 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:31.207036 10731 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:31.207089 10731 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.207124 10731 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:31.207151 10731 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.212621 10731 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:34347
I20260926 05:45:31.212698 10858 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:34347 every 8 connection(s)
I20260926 05:45:31.213022 10731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:45:31.218040 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.213800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:43415 (local address 127.23.213.254:40085)
0926 05:45:31.213855 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.213857 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.214074 (+ 217us) server_negotiation.cc:407] Connection header received
0926 05:45:31.214537 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.214538 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.214548 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.214609 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:31.215229 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.215718 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.216729 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.216914 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.216970 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.217848 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:31.217849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:31.217849 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:31.217853 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:31.217863 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:31.217866 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:31.217903 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.217964 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.217982 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:45:31.218276 10861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.213733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40085 (local address 127.23.213.194:43415)
0926 05:45:31.213944 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.213985 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.214457 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.214662 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.214665 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.214673 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:31.215165 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.215169 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.215804 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.215805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.216652 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.216655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.216992 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.216993 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.217102 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.217661 (+ 559us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:31.217661 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:31.217690 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:31.217712 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:31.217905 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:31.217908 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:31.217911 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.217971 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.218052 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":113,"threads_started":1}
I20260926 05:45:31.218739 10859 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40085
I20260926 05:45:31.218835 10859 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:31.218981 10859 heartbeater.cc:511] Master 127.23.213.254:40085 requested a full tablet report, sending...
I20260926 05:45:31.219333 10540 ts_manager.cc:194] Registered new tserver with Master: 2c1908aac48948be856e4447a9f1c741 (127.23.213.194:34347)
I20260926 05:45:31.219707 10540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.194:43415
I20260926 05:45:31.220628 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 10731
I20260926 05:45:31.220738 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:45:31.220937 24407 external_mini_cluster.cc:1428] Setting key bea6830e5d583e5c48f86daabd9edf06
I20260926 05:45:31.221874 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40085
--builtin_ntp_servers=127.23.213.212:45651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:31.301740 10862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:31.302026 10862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:31.302063 10862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:31.303236 10862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:31.303301 10862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:31.303345 10862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:45:31.304680 10862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:45651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:40085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:31.304939 10862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:31.305227 10862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:31.310216 10867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:31.310286 10868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:31.310313 10870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:31.310257 10862 server_base.cc:1034] running on GCE node
I20260926 05:45:31.310581 10862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:31.310773 10862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:31.311913 10862 hybrid_clock.cc:648] HybridClock initialized: now 1790401531311899 us; error 28 us; skew 500 ppm
I20260926 05:45:31.313130 10862 webserver.cc:460] Webserver started at http://127.23.213.195:39743/ using document root <none> and password file <none>
I20260926 05:45:31.313372 10862 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:31.313418 10862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:31.313561 10862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:31.314669 10862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "bb2f305c43714ab89789208c08f285a8"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "7ebc26fa1277436d0b7b5467bca3f1ff"
server_key_iv: "b3a3eb28f3b5a7a66d798c6778be7328"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.315016 10862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "bb2f305c43714ab89789208c08f285a8"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "7ebc26fa1277436d0b7b5467bca3f1ff"
server_key_iv: "b3a3eb28f3b5a7a66d798c6778be7328"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.316377 10862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:31.317026 10876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.317207 10862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:31.317277 10862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "bb2f305c43714ab89789208c08f285a8"
format_stamp: "Formatted at 2026-09-26 05:45:31 on dist-test-slave-1fdt"
server_key: "7ebc26fa1277436d0b7b5467bca3f1ff"
server_key_iv: "b3a3eb28f3b5a7a66d798c6778be7328"
server_key_version: "encryptionkey@0"
I20260926 05:45:31.317329 10862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:31.332607 10862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:31.332865 10862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:31.332964 10862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:31.333143 10862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:31.333418 10862 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:31.333447 10862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.333469 10862 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:31.333482 10862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:31.339169 10862 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:45835
I20260926 05:45:31.339216 10989 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:45835 every 8 connection(s)
I20260926 05:45:31.339545 10862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:45:31.344743 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.340229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:52391 (local address 127.23.213.254:40085)
0926 05:45:31.340298 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.340300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.340523 (+ 223us) server_negotiation.cc:407] Connection header received
0926 05:45:31.341066 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.341067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.341079 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.341135 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:31.341880 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.342387 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.343386 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.343605 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.343682 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.344460 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:31.344461 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:31.344461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:31.344466 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:31.344480 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:31.344484 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:31.344540 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.344636 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.344665 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:45:31.344980 10992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.340237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:40085 (local address 127.23.213.195:52391)
0926 05:45:31.340400 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.340434 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.340958 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.341216 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.341220 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.341228 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:31.341713 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.341717 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.342490 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.342491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.343294 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.343296 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.343700 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.343701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.343837 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.344320 (+ 483us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:31.344320 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:31.344347 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:31.344369 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:31.344542 (+ 173us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:31.344547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:31.344550 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.344638 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.344736 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260926 05:45:31.345453 10990 heartbeater.cc:347] Connected to a master server at 127.23.213.254:40085
I20260926 05:45:31.345546 10990 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:31.345695 10990 heartbeater.cc:511] Master 127.23.213.254:40085 requested a full tablet report, sending...
I20260926 05:45:31.346069 10540 ts_manager.cc:194] Registered new tserver with Master: bb2f305c43714ab89789208c08f285a8 (127.23.213.195:45835)
I20260926 05:45:31.346385 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 10862
I20260926 05:45:31.346468 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:45:31.346556 10540 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.195:52391
I20260926 05:45:31.346658 24407 external_mini_cluster.cc:1428] Setting key 54960cd0385d694721517e4d9689dbd5
I20260926 05:45:31.347541 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:45:31.351781 10579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.348754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.23.213.254:40085)
0926 05:45:31.348821 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.348823 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.348949 (+ 126us) server_negotiation.cc:407] Connection header received
0926 05:45:31.349023 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.349024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.349035 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.349085 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:31.349259 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.349714 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.350513 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.350721 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.350797 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.351485 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:31.351487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:31.351487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:31.351491 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:31.351505 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:31.351509 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:31.351618 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.351692 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.351702 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 05:45:31.353240 10540 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44278:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:45:31.355006 10540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:45:31.361358 10998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.357193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:34347 (local address 127.0.0.1:33634)
0926 05:45:31.357376 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.357387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.357442 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.358054 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.358055 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.358057 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:31.358102 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.358104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.358951 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.358952 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.360154 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.360156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.361054 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.361055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.361144 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.361158 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.361235 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.361291 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260926 05:45:31.361390 11001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.357312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.23.213.194:34347)
0926 05:45:31.357879 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.357884 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.357895 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:45:31.357935 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.357937 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.357952 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.358175 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:31.358224 (+ 49us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.358866 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.360242 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.360964 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.361049 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.361146 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.361245 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.361287 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":87,"threads_started":1}
I20260926 05:45:31.361683 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.357525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53988 (local address 127.23.213.195:45835)
0926 05:45:31.357690 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.357693 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.357703 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:45:31.357733 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.357734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.357776 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.357857 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:31.358090 (+ 233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.359201 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.360393 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.361064 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.361123 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.361188 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.361563 (+ 375us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.361605 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":78,"threads_started":1}
I20260926 05:45:31.361840 11002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.357345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45835 (local address 127.0.0.1:53988)
0926 05:45:31.357490 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.357499 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.357538 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.357931 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.357935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.357940 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:31.358008 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.358010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.359288 (+ 1278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.359289 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.360325 (+ 1036us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.360327 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.361222 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.361223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.361294 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.361299 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.361350 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.361390 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":6,"thread_start_us":70,"threads_started":1}
I20260926 05:45:31.362421 10999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.357041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:37377 (local address 127.0.0.1:53370)
0926 05:45:31.357323 (+ 282us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.357362 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.357425 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.357931 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.357935 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.357937 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:31.358008 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.358011 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.360391 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.360391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.361059 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.361060 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.362196 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.362197 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.362262 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.362265 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.362317 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.362358 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":249,"mutex_wait_us":5,"thread_start_us":43,"threads_started":1}
I20260926 05:45:31.362293 10793 tablet_service.cc:1467] Processing CreateTablet for tablet 825f5b55916d4387a62de6c70d4efddb (DEFAULT_TABLE table=test-table [id=c29cb50559374aca98540c89e8b6577e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:31.362612 10793 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825f5b55916d4387a62de6c70d4efddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:31.362635 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.357146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53370 (local address 127.23.213.193:37377)
0926 05:45:31.357675 (+ 529us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.357679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.357690 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:45:31.357731 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.357733 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.357749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.357857 (+ 108us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:31.358091 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.359216 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.361153 (+ 1937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.361997 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.362070 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.362160 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.362508 (+ 348us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.362554 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":132,"threads_started":1}
I20260926 05:45:31.362568 10924 tablet_service.cc:1467] Processing CreateTablet for tablet 825f5b55916d4387a62de6c70d4efddb (DEFAULT_TABLE table=test-table [id=c29cb50559374aca98540c89e8b6577e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:31.362887 10924 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825f5b55916d4387a62de6c70d4efddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:31.365144 11004 tablet_bootstrap.cc:492] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Bootstrap starting.
I20260926 05:45:31.365195 11005 tablet_bootstrap.cc:492] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Bootstrap starting.
I20260926 05:45:31.365723 11005 tablet_bootstrap.cc:654] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:31.365839 11004 tablet_bootstrap.cc:654] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:31.366080 11005 log.cc:824] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:31.366159 11004 log.cc:824] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:31.366407 10662 tablet_service.cc:1467] Processing CreateTablet for tablet 825f5b55916d4387a62de6c70d4efddb (DEFAULT_TABLE table=test-table [id=c29cb50559374aca98540c89e8b6577e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:31.366719 10662 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 825f5b55916d4387a62de6c70d4efddb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:31.366825 11004 tablet_bootstrap.cc:492] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: No bootstrap required, opened a new log
I20260926 05:45:31.366855 11005 tablet_bootstrap.cc:492] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: No bootstrap required, opened a new log
I20260926 05:45:31.366868 11004 ts_tablet_manager.cc:1397] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 05:45:31.366909 11005 ts_tablet_manager.cc:1397] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:31.368906 11004 raft_consensus.cc:348] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.369019 11004 raft_consensus.cc:374] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:31.369076 11004 raft_consensus.cc:729] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c1908aac48948be856e4447a9f1c741, State: Initialized, Role: FOLLOWER
I20260926 05:45:31.369189 11004 consensus_queue.cc:260] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.369462 10859 heartbeater.cc:503] Master 127.23.213.254:40085 was elected leader, sending a full tablet report...
I20260926 05:45:31.369459 11005 raft_consensus.cc:348] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.369613 11005 raft_consensus.cc:374] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:31.369453 11004 ts_tablet_manager.cc:1428] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 05:45:31.369648 11005 raft_consensus.cc:729] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb2f305c43714ab89789208c08f285a8, State: Initialized, Role: FOLLOWER
I20260926 05:45:31.369802 11005 consensus_queue.cc:260] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.370064 11005 ts_tablet_manager.cc:1428] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 05:45:31.370541 10990 heartbeater.cc:503] Master 127.23.213.254:40085 was elected leader, sending a full tablet report...
I20260926 05:45:31.370851 11010 tablet_bootstrap.cc:492] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6: Bootstrap starting.
I20260926 05:45:31.371402 11010 tablet_bootstrap.cc:654] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:31.371682 11010 log.cc:824] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:31.372346 11010 tablet_bootstrap.cc:492] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6: No bootstrap required, opened a new log
I20260926 05:45:31.372416 11010 ts_tablet_manager.cc:1397] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:31.374457 11010 raft_consensus.cc:348] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.374580 11010 raft_consensus.cc:374] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:31.374600 11010 raft_consensus.cc:729] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3bfa171e74b401fbf62862207af1ac6, State: Initialized, Role: FOLLOWER
I20260926 05:45:31.374730 11010 consensus_queue.cc:260] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.374974 11010 ts_tablet_manager.cc:1428] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:45:31.375041 10728 heartbeater.cc:503] Master 127.23.213.254:40085 was elected leader, sending a full tablet report...
W20260926 05:45:31.463585 10860 tablet.cc:2369] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:45:31.587176 10729 tablet.cc:2369] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:45:31.590019 10991 tablet.cc:2369] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:31.807660 11008 raft_consensus.cc:482] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:31.807792 11008 raft_consensus.cc:504] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.808252 11008 leader_election.cc:288] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3bfa171e74b401fbf62862207af1ac6 (127.23.213.193:37377), bb2f305c43714ab89789208c08f285a8 (127.23.213.195:45835)
I20260926 05:45:31.811193 11000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.808538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:43891 (local address 127.23.213.193:37377)
0926 05:45:31.808609 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.808616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.808662 (+ 46us) server_negotiation.cc:407] Connection header received
0926 05:45:31.808755 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.808758 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.808770 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.808828 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:31.808999 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.809484 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.810499 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.810847 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.810929 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.810938 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.811088 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.811122 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:45:31.811228 11013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.808458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:37377 (local address 127.23.213.194:43891)
0926 05:45:31.808584 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.808597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.808664 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.808838 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.808840 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.808842 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:31.808916 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.808919 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.809571 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.809572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.810397 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.810399 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.810938 (+ 539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.810939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.811006 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.811009 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.811071 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.811110 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":79,"threads_started":1}
I20260926 05:45:31.811244 11003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.808748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:36481 (local address 127.23.213.195:45835)
0926 05:45:31.808831 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.808833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.808843 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:45:31.808893 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.808894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.808901 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.808963 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:31.809134 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.809589 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.810531 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.810893 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.810975 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.810983 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.811131 (+ 148us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.811176 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 05:45:31.811467 11014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.808649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45835 (local address 127.23.213.194:36481)
0926 05:45:31.808771 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.808780 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:31.808822 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:31.808983 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:31.808984 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:31.808986 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:31.809062 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.809063 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.809684 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.809685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.810427 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:31.810429 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.810976 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.810977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:31.811053 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.811056 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:31.811123 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:45:31.811162 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":56,"threads_started":1}
I20260926 05:45:31.811544 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f5b55916d4387a62de6c70d4efddb" candidate_uuid: "2c1908aac48948be856e4447a9f1c741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bfa171e74b401fbf62862207af1ac6" is_pre_election: true
I20260926 05:45:31.811709 10682 raft_consensus.cc:2454] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c1908aac48948be856e4447a9f1c741 in term 0.
I20260926 05:45:31.811836 10944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f5b55916d4387a62de6c70d4efddb" candidate_uuid: "2c1908aac48948be856e4447a9f1c741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2f305c43714ab89789208c08f285a8" is_pre_election: true
I20260926 05:45:31.811947 10747 leader_election.cc:302] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c1908aac48948be856e4447a9f1c741, d3bfa171e74b401fbf62862207af1ac6; no voters:
I20260926 05:45:31.811987 10944 raft_consensus.cc:2454] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c1908aac48948be856e4447a9f1c741 in term 0.
I20260926 05:45:31.812062 11008 raft_consensus.cc:2781] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:45:31.812140 11008 raft_consensus.cc:482] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:45:31.812175 11008 raft_consensus.cc:3037] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:31.812848 11008 raft_consensus.cc:504] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.812942 11008 leader_election.cc:288] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [CANDIDATE]: Term 1 election: Requested vote from peers d3bfa171e74b401fbf62862207af1ac6 (127.23.213.193:37377), bb2f305c43714ab89789208c08f285a8 (127.23.213.195:45835)
I20260926 05:45:31.813124 10682 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f5b55916d4387a62de6c70d4efddb" candidate_uuid: "2c1908aac48948be856e4447a9f1c741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3bfa171e74b401fbf62862207af1ac6"
I20260926 05:45:31.813153 10944 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "825f5b55916d4387a62de6c70d4efddb" candidate_uuid: "2c1908aac48948be856e4447a9f1c741" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bb2f305c43714ab89789208c08f285a8"
I20260926 05:45:31.813212 10682 raft_consensus.cc:3037] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:31.813261 10944 raft_consensus.cc:3037] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:31.814189 10682 raft_consensus.cc:2454] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c1908aac48948be856e4447a9f1c741 in term 1.
I20260926 05:45:31.814189 10944 raft_consensus.cc:2454] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c1908aac48948be856e4447a9f1c741 in term 1.
I20260926 05:45:31.814352 10747 leader_election.cc:302] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c1908aac48948be856e4447a9f1c741, d3bfa171e74b401fbf62862207af1ac6; no voters:
I20260926 05:45:31.814455 11008 raft_consensus.cc:2781] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:31.814599 11008 raft_consensus.cc:686] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 1 LEADER]: Becoming Leader. State: Replica: 2c1908aac48948be856e4447a9f1c741, State: Running, Role: LEADER
I20260926 05:45:31.814700 11008 consensus_queue.cc:237] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } }
I20260926 05:45:31.815485 10540 catalog_manager.cc:5495] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c1908aac48948be856e4447a9f1c741 (127.23.213.194). New cstate: current_term: 1 leader_uuid: "2c1908aac48948be856e4447a9f1c741" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2c1908aac48948be856e4447a9f1c741" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 34347 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 } health_report { overall_health: UNKNOWN } } }
I20260926 05:45:31.910903 11019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.908335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.23.213.194:34347)
0926 05:45:31.908460 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:31.908462 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:31.908471 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:45:31.908538 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:31.908539 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:31.908565 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:31.908652 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:45:31.908804 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.909289 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.910147 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:31.910348 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:31.910419 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:31.910561 (+ 142us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:45:31.910660 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:45:31.910780 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:31.910811 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 05:45:31.910820 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":55,"threads_started":1}
I20260926 05:45:31.912763 10944 raft_consensus.cc:1261] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 1 FOLLOWER]: Refusing update from remote peer 2c1908aac48948be856e4447a9f1c741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:31.912763 10682 raft_consensus.cc:1261] T 825f5b55916d4387a62de6c70d4efddb P d3bfa171e74b401fbf62862207af1ac6 [term 1 FOLLOWER]: Refusing update from remote peer 2c1908aac48948be856e4447a9f1c741: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:31.913144 11015 consensus_queue.cc:1035] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bb2f305c43714ab89789208c08f285a8" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45835 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:31.913304 11008 consensus_queue.cc:1035] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3bfa171e74b401fbf62862207af1ac6" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 37377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:31.916112 11023 mvcc.cc:205] Tried to move back new op lower bound from 7333484674711523328 to 7333484674313588736. Current Snapshot: MvccSnapshot[applied={T|T < 7333484674711523328}]
I20260926 05:45:31.916141 11024 mvcc.cc:205] Tried to move back new op lower bound from 7333484674711523328 to 7333484674313588736. Current Snapshot: MvccSnapshot[applied={T|T < 7333484674711523328}]
I20260926 05:45:31.918270 10539 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:31.918363 10539 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44278:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:31.919128 10539 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 411f86051ce14f758d27a4136b124688: Sending DeleteTablet for 3 replicas of tablet 825f5b55916d4387a62de6c70d4efddb
I20260926 05:45:31.919507 10662 tablet_service.cc:1514] Processing DeleteTablet for tablet 825f5b55916d4387a62de6c70d4efddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:31 UTC) from {username='slave'} at 127.0.0.1:53370
I20260926 05:45:31.919479 10793 tablet_service.cc:1514] Processing DeleteTablet for tablet 825f5b55916d4387a62de6c70d4efddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:31 UTC) from {username='slave'} at 127.0.0.1:33634
I20260926 05:45:31.919778 10924 tablet_service.cc:1514] Processing DeleteTablet for tablet 825f5b55916d4387a62de6c70d4efddb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:31 UTC) from {username='slave'} at 127.0.0.1:53988
I20260926 05:45:31.919857 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 10600
I20260926 05:45:31.920429 11033 tablet_replica.cc:330] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: stopping tablet replica
I20260926 05:45:31.920531 11033 raft_consensus.cc:2229] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:45:31.920648 11033 raft_consensus.cc:2258] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:31.921046 11033 ts_tablet_manager.cc:1905] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:31.921890 11034 tablet_replica.cc:330] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: stopping tablet replica
I20260926 05:45:31.921990 11034 raft_consensus.cc:2229] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:31.922093 11034 raft_consensus.cc:2258] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:31.922443 11034 ts_tablet_manager.cc:1905] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:31.923002 11033 ts_tablet_manager.cc:1918] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:31.923077 11033 log.cc:1196] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/825f5b55916d4387a62de6c70d4efddb
I20260926 05:45:31.923357 11033 ts_tablet_manager.cc:1939] T 825f5b55916d4387a62de6c70d4efddb P 2c1908aac48948be856e4447a9f1c741: Deleting consensus metadata
I20260926 05:45:31.923771 10525 catalog_manager.cc:4841] TS 2c1908aac48948be856e4447a9f1c741 (127.23.213.194:34347): tablet 825f5b55916d4387a62de6c70d4efddb (table test-table [id=c29cb50559374aca98540c89e8b6577e]) successfully deleted
I20260926 05:45:31.924187 11034 ts_tablet_manager.cc:1918] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:31.924248 11034 log.cc:1196] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/825f5b55916d4387a62de6c70d4efddb
I20260926 05:45:31.924419 11034 ts_tablet_manager.cc:1939] T 825f5b55916d4387a62de6c70d4efddb P bb2f305c43714ab89789208c08f285a8: Deleting consensus metadata
I20260926 05:45:31.925885 10525 catalog_manager.cc:4841] TS bb2f305c43714ab89789208c08f285a8 (127.23.213.195:45835): tablet 825f5b55916d4387a62de6c70d4efddb (table test-table [id=c29cb50559374aca98540c89e8b6577e]) successfully deleted
W20260926 05:45:31.927778 10525 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:37377 (error 108)
I20260926 05:45:31.928347 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 10731
I20260926 05:45:31.928372 11036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:31.928176 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:37377 (local address 127.0.0.1:53374)
0926 05:45:31.928272 (+ 96us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:31.928330 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:37377: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":60,"threads_started":1}
W20260926 05:45:31.929946 10525 catalog_manager.cc:4584] TS d3bfa171e74b401fbf62862207af1ac6 (127.23.213.193:37377): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 825f5b55916d4387a62de6c70d4efddb: Network error: Client connection negotiation failed: client connection to 127.23.213.193:37377: connect: Connection refused (error 111)
I20260926 05:45:31.932754 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 10862
I20260926 05:45:31.937414 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 10509
2026-09-26T05:45:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (1189 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:31 dist-test-slave-1fdt krb5kdc[11039](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:31 dist-test-slave-1fdt krb5kdc[11039](info): set up 2 sockets
Sep 26 05:45:31 dist-test-slave-1fdt krb5kdc[11039](info): commencing operation
krb5kdc: starting...
W20260926 05:45:32.969601 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:33.088707 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41383
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:41383
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:33.168052 11052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:33.168308 11052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:33.168349 11052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:33.169490 11052 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:33.169562 11052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:33.169587 11052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:33.169608 11052 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:33.169638 11052 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:33.171144 11052 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:41383
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:33.171418 11052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:33.171641 11052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:45:33.176580 11052 server_base.cc:1034] running on GCE node
W20260926 05:45:33.176535 11058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.176550 11060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.176544 11057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:33.177016 11052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:33.177390 11052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:33.178601 11052 hybrid_clock.cc:648] HybridClock initialized: now 1790401533178584 us; error 38 us; skew 500 ppm
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:33.181202 11052 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:33.181660 11052 webserver.cc:460] Webserver started at http://127.23.213.254:34337/ using document root <none> and password file <none>
I20260926 05:45:33.181927 11052 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:33.182006 11052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:33.182179 11052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:33.183336 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "1b81cd432805410b91ed3d3344afc9f3"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "f7d5825938cbd44c322d040742becdf5"
server_key_iv: "4124c6623dbc79bbe0c3454db4e30bf6"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.183718 11052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "1b81cd432805410b91ed3d3344afc9f3"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "f7d5825938cbd44c322d040742becdf5"
server_key_iv: "4124c6623dbc79bbe0c3454db4e30bf6"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.185189 11052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:33.186022 11067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.186218 11052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:33.186332 11052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "1b81cd432805410b91ed3d3344afc9f3"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "f7d5825938cbd44c322d040742becdf5"
server_key_iv: "4124c6623dbc79bbe0c3454db4e30bf6"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.186409 11052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:33.205367 11052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:33.207511 11052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:33.207701 11052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:33.211737 11052 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:41383
I20260926 05:45:33.211774 11119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:41383 every 8 connection(s)
I20260926 05:45:33.212142 11052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:33.213007 11120 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:33.213552 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11052
I20260926 05:45:33.213702 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:33.213954 24407 external_mini_cluster.cc:1428] Setting key ddffa87312e1fe6618072e2d6894e7df
I20260926 05:45:33.215865 11120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3: Bootstrap starting.
I20260926 05:45:33.216424 11120 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:33.216724 11120 log.cc:824] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:33.217381 11120 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3: No bootstrap required, opened a new log
I20260926 05:45:33.219409 11120 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81cd432805410b91ed3d3344afc9f3" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41383 } }
I20260926 05:45:33.219561 11120 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:33.219624 11120 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b81cd432805410b91ed3d3344afc9f3, State: Initialized, Role: FOLLOWER
I20260926 05:45:33.219790 11120 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81cd432805410b91ed3d3344afc9f3" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41383 } }
I20260926 05:45:33.219887 11120 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:33.219929 11120 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:33.219977 11120 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:33.220512 11120 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81cd432805410b91ed3d3344afc9f3" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41383 } }
I20260926 05:45:33.220630 11120 leader_election.cc:302] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b81cd432805410b91ed3d3344afc9f3; no voters:
I20260926 05:45:33.220816 11120 leader_election.cc:288] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:33.220867 11125 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:33.221020 11125 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b81cd432805410b91ed3d3344afc9f3, State: Running, Role: LEADER
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:33.221082 11120 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:33.221154 11125 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81cd432805410b91ed3d3344afc9f3" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41383 } }
I20260926 05:45:33.221495 11125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b81cd432805410b91ed3d3344afc9f3. Latest consensus state: current_term: 1 leader_uuid: "1b81cd432805410b91ed3d3344afc9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81cd432805410b91ed3d3344afc9f3" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41383 } } }
I20260926 05:45:33.221658 11125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:33.221684 11126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b81cd432805410b91ed3d3344afc9f3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81cd432805410b91ed3d3344afc9f3" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41383 } } }
I20260926 05:45:33.221850 11126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:33.221994 11134 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:33.222580 11134 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:33.225564 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.214520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32782 (local address 127.23.213.254:41383)
0926 05:45:33.214707 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.214717 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.214750 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:45:33.215233 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.215240 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.215289 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.215449 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:33.216197 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.217234 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.218223 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.218646 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.218787 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.221655 (+ 2868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:33.221660 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:33.221660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:33.221673 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:33.223892 (+ 2219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.224346 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.224348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.224349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.224442 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.224704 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.224705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.224705 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.224953 (+ 248us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:33.225178 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.225247 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.225388 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":78,"threads_started":1}
I20260926 05:45:33.226469 11134 catalog_manager.cc:1347] Generated new cluster ID: d8cf20c6ca2e427a9ba608bcc27c5699
I20260926 05:45:33.226516 11134 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:33.234848 11134 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:33.235535 11134 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:33.242650 11134 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3: Generated new TSK 0
I20260926 05:45:33.242856 11134 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.23.213.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
Entry for principal kudu/127.23.213.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab.
I20260926 05:45:33.270478 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41383
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:33.353121 11146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:33.353341 11146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:33.353367 11146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:33.354701 11146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:33.354748 11146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:33.354801 11146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:45:33.356390 11146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.193.keytab
--principal=kudu/127.23.213.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11146
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:33.356644 11146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:33.356915 11146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:33.362162 11154 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.362167 11152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.362217 11151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:33.362479 11146 server_base.cc:1034] running on GCE node
I20260926 05:45:33.362690 11146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:33.362958 11146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:33.364101 11146 hybrid_clock.cc:648] HybridClock initialized: now 1790401533364085 us; error 28 us; skew 500 ppm
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:33.366676 11146 init.cc:377] Logged in from keytab as kudu/127.23.213.193@KRBTEST.COM (short username kudu)
I20260926 05:45:33.367095 11146 webserver.cc:460] Webserver started at http://127.23.213.193:44063/ using document root <none> and password file <none>
I20260926 05:45:33.367347 11146 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:33.367394 11146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:33.367560 11146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:33.368598 11146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "87bfe270c0d142f6accbcc6339d6588b"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "b92fd7df4b29bbe4371a26e2615b5fea"
server_key_iv: "ae2e470bfd7ef9949dee0fbb84ab63a5"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.368954 11146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "87bfe270c0d142f6accbcc6339d6588b"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "b92fd7df4b29bbe4371a26e2615b5fea"
server_key_iv: "ae2e470bfd7ef9949dee0fbb84ab63a5"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.370370 11146 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:33.371169 11161 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.371413 11146 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:33.371520 11146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "87bfe270c0d142f6accbcc6339d6588b"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "b92fd7df4b29bbe4371a26e2615b5fea"
server_key_iv: "ae2e470bfd7ef9949dee0fbb84ab63a5"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.371595 11146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:33.383342 11146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:33.385362 11146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:33.385555 11146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:33.385819 11146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:33.386128 11146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:33.386183 11146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.386221 11146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:33.386276 11146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.391940 11146 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:45695
I20260926 05:45:33.392038 11274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:45695 every 8 connection(s)
I20260926 05:45:33.392372 11146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:45:33.395254 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11146
I20260926 05:45:33.395359 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:45:33.395545 24407 external_mini_cluster.cc:1428] Setting key 9305fdf5610391ce1d300cc84b7175c0
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.193@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:33.402064 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.393120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:59833 (local address 127.23.213.254:41383)
0926 05:45:33.393183 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.393185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.393382 (+ 197us) server_negotiation.cc:407] Connection header received
0926 05:45:33.394002 (+ 620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.394003 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.394013 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.394069 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:33.394960 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.395751 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.396836 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.397169 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.397273 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.399273 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:33.399274 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:33.399275 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:33.399281 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:33.400639 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.401037 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.401038 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.401038 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.401102 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.401349 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.401350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.401350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.401468 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:33.401537 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.401877 (+ 340us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.401964 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:45:33.402252 11277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.393089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41383 (local address 127.23.213.193:59833)
0926 05:45:33.393248 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.393306 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.393907 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.394127 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.394130 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.394691 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:33.394883 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.394888 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.395868 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.395869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.396752 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.396755 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.397258 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.397259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.397381 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.397906 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:33.397914 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:33.399169 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:33.400722 (+ 1553us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:33.400729 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:33.400731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:33.400953 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:33.401166 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:33.401167 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:33.401167 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:33.401281 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:33.401540 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:33.401547 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:33.401681 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.401849 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.402015 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260926 05:45:33.402981 11275 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41383
I20260926 05:45:33.403087 11275 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:33.403259 11275 heartbeater.cc:511] Master 127.23.213.254:41383 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.23.213.194@KRBTEST.COM; defaulting to no policy
I20260926 05:45:33.403823 11084 ts_manager.cc:194] Registered new tserver with Master: 87bfe270c0d142f6accbcc6339d6588b (127.23.213.193:45695)
I20260926 05:45:33.404462 11084 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.193@KRBTEST.COM'} at 127.23.213.193:59833
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
Entry for principal kudu/127.23.213.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab.
I20260926 05:45:33.421025 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41383
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:33.502329 11280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:33.502638 11280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:33.502676 11280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:33.504522 11280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:33.504597 11280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:33.504642 11280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:45:33.506245 11280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.194.keytab
--principal=kudu/127.23.213.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:33.506480 11280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:33.506732 11280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:33.511588 11286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.511629 11285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.511587 11288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:33.511890 11280 server_base.cc:1034] running on GCE node
I20260926 05:45:33.512101 11280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:33.512343 11280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:33.513482 11280 hybrid_clock.cc:648] HybridClock initialized: now 1790401533513453 us; error 55 us; skew 500 ppm
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:33.515940 11280 init.cc:377] Logged in from keytab as kudu/127.23.213.194@KRBTEST.COM (short username kudu)
I20260926 05:45:33.516415 11280 webserver.cc:460] Webserver started at http://127.23.213.194:45551/ using document root <none> and password file <none>
I20260926 05:45:33.516690 11280 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:33.516764 11280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:33.516896 11280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:33.518002 11280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "e47f2ce7ca58405a8d0afcfaec578182"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "0ec83fee9d1fe0349b1c2aea199315f8"
server_key_iv: "0270a9bde5c19c218e1beeb587378da3"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.518352 11280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "e47f2ce7ca58405a8d0afcfaec578182"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "0ec83fee9d1fe0349b1c2aea199315f8"
server_key_iv: "0270a9bde5c19c218e1beeb587378da3"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.519649 11280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:33.520452 11295 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.520644 11280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:33.520741 11280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "e47f2ce7ca58405a8d0afcfaec578182"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "0ec83fee9d1fe0349b1c2aea199315f8"
server_key_iv: "0270a9bde5c19c218e1beeb587378da3"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.520828 11280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:33.538262 11280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:33.540206 11280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:33.540375 11280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:33.540617 11280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:33.540946 11280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:33.540998 11280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.541034 11280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:33.541062 11280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.546715 11280 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:35937
I20260926 05:45:33.546787 11408 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:35937 every 8 connection(s)
I20260926 05:45:33.547117 11280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.194@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:33.555932 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.547886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:51877 (local address 127.23.213.254:41383)
0926 05:45:33.547950 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.547952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.548151 (+ 199us) server_negotiation.cc:407] Connection header received
0926 05:45:33.548619 (+ 468us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.548620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.548629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.548692 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:33.549601 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.550190 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.551140 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.551332 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.551388 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.553332 (+ 1944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:33.553333 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:33.553334 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:33.553339 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:33.554783 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.555118 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.555119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.555119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.555169 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.555377 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.555378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.555378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.555471 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:33.555517 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.555748 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.555833 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:45:33.555923 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11280
I20260926 05:45:33.555994 11411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.547871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41383 (local address 127.23.213.194:51877)
0926 05:45:33.548026 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.548068 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.548539 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.548753 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.548757 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.549306 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:33.549502 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.549507 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.550270 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.550270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.551058 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.551059 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.551405 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.551405 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.551502 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.551932 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:33.551940 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:33.553248 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:33.554847 (+ 1599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:33.554851 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:33.554853 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:33.555049 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:33.555223 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:33.555224 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:33.555224 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:33.555312 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:33.555519 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:33.555522 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:33.555626 (+ 104us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.555701 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.555819 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":87,"threads_started":1}
I20260926 05:45:33.556025 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:45:33.556200 24407 external_mini_cluster.cc:1428] Setting key 24e215c4b735ca1eb13600c033b93fd2
I20260926 05:45:33.556736 11409 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41383
I20260926 05:45:33.556851 11409 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:33.557036 11409 heartbeater.cc:511] Master 127.23.213.254:41383 requested a full tablet report, sending...
I20260926 05:45:33.557427 11083 ts_manager.cc:194] Registered new tserver with Master: e47f2ce7ca58405a8d0afcfaec578182 (127.23.213.194:35937)
I20260926 05:45:33.557885 11083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.194@KRBTEST.COM'} at 127.23.213.194:51877
WARNING: no policy specified for kudu/127.23.213.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
Entry for principal kudu/127.23.213.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab.
I20260926 05:45:33.581246 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41383
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:41819
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:33.661375 11414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:33.661571 11414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:33.661593 11414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:33.662792 11414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:33.662837 11414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:33.662861 11414 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:45:33.664826 11414 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:41819
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.195.keytab
--principal=kudu/127.23.213.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:41383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:33.665126 11414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:33.665374 11414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:33.670225 11419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.670241 11420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:33.670238 11422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:33.670674 11414 server_base.cc:1034] running on GCE node
I20260926 05:45:33.670892 11414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:33.671128 11414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:33.672356 11414 hybrid_clock.cc:648] HybridClock initialized: now 1790401533672338 us; error 25 us; skew 500 ppm
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:33.675066 11414 init.cc:377] Logged in from keytab as kudu/127.23.213.195@KRBTEST.COM (short username kudu)
I20260926 05:45:33.675582 11414 webserver.cc:460] Webserver started at http://127.23.213.195:37705/ using document root <none> and password file <none>
I20260926 05:45:33.675833 11414 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:33.675911 11414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:33.676115 11414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:33.677227 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "cceac09224e44e929841f02ce4553950"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "f9c7afa62ebb6584a273ee31ee7ff1cc"
server_key_iv: "96ffbdffa4f8339394c6adb4459e8348"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.677608 11414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "cceac09224e44e929841f02ce4553950"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "f9c7afa62ebb6584a273ee31ee7ff1cc"
server_key_iv: "96ffbdffa4f8339394c6adb4459e8348"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.679152 11414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:33.679901 11429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.680073 11414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:33.680192 11414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "cceac09224e44e929841f02ce4553950"
format_stamp: "Formatted at 2026-09-26 05:45:33 on dist-test-slave-1fdt"
server_key: "f9c7afa62ebb6584a273ee31ee7ff1cc"
server_key_iv: "96ffbdffa4f8339394c6adb4459e8348"
server_key_version: "encryptionkey@0"
I20260926 05:45:33.680269 11414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:33.689929 11414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:33.692203 11414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:33.692373 11414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:33.692593 11414 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:33.692893 11414 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:33.692926 11414 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.692970 11414 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:33.692996 11414 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:33.699152 11414 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:42761
I20260926 05:45:33.699250 11542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:42761 every 8 connection(s)
I20260926 05:45:33.699563 11414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
Sep 26 05:45:33 dist-test-slave-1fdt krb5kdc[11039](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401533, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.195@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:33.706058 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11414
I20260926 05:45:33.706162 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:45:33.706351 24407 external_mini_cluster.cc:1428] Setting key d3ed858c04914fae8859c41bc455dbe6
I20260926 05:45:33.708616 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.700270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:48437 (local address 127.23.213.254:41383)
0926 05:45:33.700352 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.700354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.700528 (+ 174us) server_negotiation.cc:407] Connection header received
0926 05:45:33.700992 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.700993 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.701003 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.701052 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:33.701976 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.702471 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.703459 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.703638 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.703686 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.705639 (+ 1953us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:33.705640 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:33.705641 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:33.705648 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:33.707330 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.707690 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.707691 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.707691 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.707747 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.707995 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.707996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.707996 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.708113 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:33.708188 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.708452 (+ 264us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.708527 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 05:45:33.708814 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.700230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:41383 (local address 127.23.213.195:48437)
0926 05:45:33.700405 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.700439 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.700913 (+ 474us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.701102 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.701105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.701635 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:33.701903 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.701909 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.702545 (+ 636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.702546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.703387 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.703389 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.703713 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.703713 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.703808 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.704240 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:33.704247 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:33.705531 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:33.707394 (+ 1863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:33.707399 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:33.707402 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:33.707613 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:33.707824 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:33.707825 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:33.707825 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:33.707926 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:33.708180 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:33.708186 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:33.708321 (+ 135us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.708430 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.708580 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260926 05:45:33.709277 11543 heartbeater.cc:347] Connected to a master server at 127.23.213.254:41383
I20260926 05:45:33.709371 11543 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:33.709503 11543 heartbeater.cc:511] Master 127.23.213.254:41383 requested a full tablet report, sending...
I20260926 05:45:33.709861 11083 ts_manager.cc:194] Registered new tserver with Master: cceac09224e44e929841f02ce4553950 (127.23.213.195:42761)
I20260926 05:45:33.710215 11083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.23.213.195@KRBTEST.COM'} at 127.23.213.195:48437
I20260926 05:45:33.718217 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:45:33.725653 11123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.719530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32790 (local address 127.23.213.254:41383)
0926 05:45:33.719615 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.719617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.719697 (+ 80us) server_negotiation.cc:407] Connection header received
0926 05:45:33.719765 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.719766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.719776 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.719831 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:33.720405 (+ 574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.720970 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.721792 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.722016 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.722088 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.723297 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:33.723298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:33.723298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:33.723304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:33.724657 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.725006 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.725006 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.725007 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.725057 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:33.725202 (+ 145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:33.725203 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:33.725203 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:33.725326 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:33.725396 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.725539 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.725594 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 05:45:33.727102 11083 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32790:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:45:33.728597 11083 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:45:33.734841 11551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.730637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:42761 (local address 127.0.0.1:37308)
0926 05:45:33.730794 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.730836 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.730902 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.731104 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.731108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.731110 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:33.731178 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.731180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.732477 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.732478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.733647 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.733649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.734513 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.734514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.734601 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.734615 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.734687 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.734755 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":59,"thread_start_us":65,"threads_started":1}
I20260926 05:45:33.734862 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.730712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37308 (local address 127.23.213.195:42761)
0926 05:45:33.730885 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.730890 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.730901 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:45:33.730981 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.730983 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.731001 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.731091 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:33.731262 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.732366 (+ 1104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.733730 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.734420 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.734499 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.734554 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.734720 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.734765 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":70,"threads_started":1}
I20260926 05:45:33.735028 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.731015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.23.213.193:45695)
0926 05:45:33.731186 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.731191 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.731262 (+ 71us) server_negotiation.cc:407] Connection header received
0926 05:45:33.731350 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.731353 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.731369 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.731450 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:33.731644 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.732355 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.733998 (+ 1643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.734533 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.734603 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.734655 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.734879 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.734925 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":92,"threads_started":1}
I20260926 05:45:33.735131 11552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.730680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:35937 (local address 127.0.0.1:38034)
0926 05:45:33.730982 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.730991 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.731028 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.731621 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.731622 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.731624 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:33.731659 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.731660 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.732836 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.732837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.733942 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.733944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.734843 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.734845 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.734964 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.734968 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.734999 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.735040 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"thread_start_us":55,"threads_started":1}
I20260926 05:45:33.735260 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.730839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:45695 (local address 127.0.0.1:37372)
0926 05:45:33.731197 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.731207 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.731269 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.731522 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.731523 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.731525 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:33.731571 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.731572 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.732459 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.732460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.733901 (+ 1441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.733904 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.734687 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.734688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.734745 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.734749 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.734794 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.734826 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":345,"mutex_wait_us":78,"thread_start_us":35,"threads_started":1}
I20260926 05:45:33.735425 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.730820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38034 (local address 127.23.213.194:35937)
0926 05:45:33.731399 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.731415 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.731424 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:45:33.731527 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.731529 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.731545 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.731614 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:33.731713 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.732721 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.734021 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.734747 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.735214 (+ 467us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.735269 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.735289 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.735320 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":80,"threads_started":1}
I20260926 05:45:33.736074 11477 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9436a5f7ea429e8d1700c0de0dabc5 (DEFAULT_TABLE table=test-table [id=c8b2b15bba104b168aca3a95ee1da81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:33.736442 11477 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9436a5f7ea429e8d1700c0de0dabc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:33.736354 11209 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9436a5f7ea429e8d1700c0de0dabc5 (DEFAULT_TABLE table=test-table [id=c8b2b15bba104b168aca3a95ee1da81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:33.736675 11209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9436a5f7ea429e8d1700c0de0dabc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:33.738104 11343 tablet_service.cc:1467] Processing CreateTablet for tablet 8b9436a5f7ea429e8d1700c0de0dabc5 (DEFAULT_TABLE table=test-table [id=c8b2b15bba104b168aca3a95ee1da81d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:33.738435 11343 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b9436a5f7ea429e8d1700c0de0dabc5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:33.740017 11558 tablet_bootstrap.cc:492] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b: Bootstrap starting.
I20260926 05:45:33.740204 11557 tablet_bootstrap.cc:492] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Bootstrap starting.
I20260926 05:45:33.740863 11557 tablet_bootstrap.cc:654] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:33.741447 11557 log.cc:824] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:33.741446 11559 tablet_bootstrap.cc:492] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Bootstrap starting.
I20260926 05:45:33.742106 11559 tablet_bootstrap.cc:654] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:33.742311 11558 tablet_bootstrap.cc:654] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:33.742312 11557 tablet_bootstrap.cc:492] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: No bootstrap required, opened a new log
I20260926 05:45:33.742401 11557 ts_tablet_manager.cc:1397] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 05:45:33.742682 11559 log.cc:824] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:33.743024 11558 log.cc:824] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:33.743546 11559 tablet_bootstrap.cc:492] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: No bootstrap required, opened a new log
I20260926 05:45:33.743614 11559 ts_tablet_manager.cc:1397] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 05:45:33.744493 11557 raft_consensus.cc:348] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.744673 11557 raft_consensus.cc:374] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:33.744709 11557 raft_consensus.cc:729] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cceac09224e44e929841f02ce4553950, State: Initialized, Role: FOLLOWER
I20260926 05:45:33.744828 11557 consensus_queue.cc:260] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.745100 11557 ts_tablet_manager.cc:1428] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:45:33.745126 11543 heartbeater.cc:503] Master 127.23.213.254:41383 was elected leader, sending a full tablet report...
I20260926 05:45:33.745592 11559 raft_consensus.cc:348] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.745791 11559 raft_consensus.cc:374] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:33.745829 11559 raft_consensus.cc:729] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e47f2ce7ca58405a8d0afcfaec578182, State: Initialized, Role: FOLLOWER
I20260926 05:45:33.745940 11559 consensus_queue.cc:260] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.746213 11559 ts_tablet_manager.cc:1428] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 05:45:33.746232 11409 heartbeater.cc:503] Master 127.23.213.254:41383 was elected leader, sending a full tablet report...
I20260926 05:45:33.746337 11558 tablet_bootstrap.cc:492] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b: No bootstrap required, opened a new log
I20260926 05:45:33.746389 11558 ts_tablet_manager.cc:1397] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 05:45:33.747691 11558 raft_consensus.cc:348] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.747840 11558 raft_consensus.cc:374] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:33.747885 11558 raft_consensus.cc:729] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87bfe270c0d142f6accbcc6339d6588b, State: Initialized, Role: FOLLOWER
I20260926 05:45:33.748014 11558 consensus_queue.cc:260] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.748301 11558 ts_tablet_manager.cc:1428] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 05:45:33.748313 11275 heartbeater.cc:503] Master 127.23.213.254:41383 was elected leader, sending a full tablet report...
W20260926 05:45:33.797720 11410 tablet.cc:2369] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:45:33.893308 11276 tablet.cc:2369] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:45:33.950156 11544 tablet.cc:2369] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:33.967658 11563 raft_consensus.cc:482] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:33.967865 11563 raft_consensus.cc:504] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.968262 11563 leader_election.cc:288] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 87bfe270c0d142f6accbcc6339d6588b (127.23.213.193:45695), e47f2ce7ca58405a8d0afcfaec578182 (127.23.213.194:35937)
I20260926 05:45:33.971155 11545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.968469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:45695 (local address 127.23.213.195:40609)
0926 05:45:33.968526 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.968538 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.968587 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.968766 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.968767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.968770 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:33.968846 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.968848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.969525 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.969526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.970340 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.970342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.970903 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.970904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.970983 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.970988 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.971041 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.971089 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 05:45:33.971179 11556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.968543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:40609 (local address 127.23.213.193:45695)
0926 05:45:33.968590 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.968593 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.968630 (+ 37us) server_negotiation.cc:407] Connection header received
0926 05:45:33.968664 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.968665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.968675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.968732 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:33.968923 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.969447 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.970424 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.970828 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.970882 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.970890 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.971060 (+ 170us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.971097 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:45:33.971336 11566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.968472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:35937 (local address 127.23.213.195:49109)
0926 05:45:33.968691 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:33.968701 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:33.968759 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:33.968917 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:33.968918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:33.968920 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:33.968984 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.968985 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.969645 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.969646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.970459 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:33.970461 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.971065 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.971066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:33.971141 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.971145 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:33.971202 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 05:45:33.971239 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"mutex_wait_us":65,"thread_start_us":95,"threads_started":1}
I20260926 05:45:33.971359 11555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:33.968573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:49109 (local address 127.23.213.194:35937)
0926 05:45:33.968615 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:33.968617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:33.968773 (+ 156us) server_negotiation.cc:407] Connection header received
0926 05:45:33.968810 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:33.968811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:33.968822 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:33.968886 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:33.969054 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.969564 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.970548 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:33.970977 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:33.971051 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:33.971058 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:33.971226 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 05:45:33.971272 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:45:33.971554 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9436a5f7ea429e8d1700c0de0dabc5" candidate_uuid: "cceac09224e44e929841f02ce4553950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87bfe270c0d142f6accbcc6339d6588b" is_pre_election: true
I20260926 05:45:33.971777 11229 raft_consensus.cc:2454] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cceac09224e44e929841f02ce4553950 in term 0.
I20260926 05:45:33.971765 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9436a5f7ea429e8d1700c0de0dabc5" candidate_uuid: "cceac09224e44e929841f02ce4553950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47f2ce7ca58405a8d0afcfaec578182" is_pre_election: true
I20260926 05:45:33.971913 11363 raft_consensus.cc:2454] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cceac09224e44e929841f02ce4553950 in term 0.
I20260926 05:45:33.972043 11431 leader_election.cc:302] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87bfe270c0d142f6accbcc6339d6588b, cceac09224e44e929841f02ce4553950; no voters:
I20260926 05:45:33.972194 11563 raft_consensus.cc:2781] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:45:33.972252 11563 raft_consensus.cc:482] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:45:33.972278 11563 raft_consensus.cc:3037] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:33.972999 11563 raft_consensus.cc:504] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.973140 11563 leader_election.cc:288] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [CANDIDATE]: Term 1 election: Requested vote from peers 87bfe270c0d142f6accbcc6339d6588b (127.23.213.193:45695), e47f2ce7ca58405a8d0afcfaec578182 (127.23.213.194:35937)
I20260926 05:45:33.973290 11229 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9436a5f7ea429e8d1700c0de0dabc5" candidate_uuid: "cceac09224e44e929841f02ce4553950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "87bfe270c0d142f6accbcc6339d6588b"
I20260926 05:45:33.973368 11229 raft_consensus.cc:3037] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:33.973340 11363 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b9436a5f7ea429e8d1700c0de0dabc5" candidate_uuid: "cceac09224e44e929841f02ce4553950" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e47f2ce7ca58405a8d0afcfaec578182"
I20260926 05:45:33.973443 11363 raft_consensus.cc:3037] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:33.974189 11229 raft_consensus.cc:2454] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cceac09224e44e929841f02ce4553950 in term 1.
I20260926 05:45:33.974257 11363 raft_consensus.cc:2454] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cceac09224e44e929841f02ce4553950 in term 1.
I20260926 05:45:33.974440 11430 leader_election.cc:302] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cceac09224e44e929841f02ce4553950, e47f2ce7ca58405a8d0afcfaec578182; no voters:
I20260926 05:45:33.974550 11563 raft_consensus.cc:2781] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:33.974704 11563 raft_consensus.cc:686] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 1 LEADER]: Becoming Leader. State: Replica: cceac09224e44e929841f02ce4553950, State: Running, Role: LEADER
I20260926 05:45:33.974817 11563 consensus_queue.cc:237] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } }
I20260926 05:45:33.975626 11083 catalog_manager.cc:5495] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 reported cstate change: term changed from 0 to 1, leader changed from <none> to cceac09224e44e929841f02ce4553950 (127.23.213.195). New cstate: current_term: 1 leader_uuid: "cceac09224e44e929841f02ce4553950" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cceac09224e44e929841f02ce4553950" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 42761 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 } health_report { overall_health: UNKNOWN } } }
I20260926 05:45:34.016731 11553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.014483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37310 (local address 127.23.213.195:42761)
0926 05:45:34.014545 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.014547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.014556 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:45:34.014597 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.014598 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.014623 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.014691 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:45:34.014823 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.015306 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.016010 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.016198 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.016248 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.016379 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:45:34.016488 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:45:34.016547 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.016578 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.016595 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:45:34.018630 11363 raft_consensus.cc:1261] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 1 FOLLOWER]: Refusing update from remote peer cceac09224e44e929841f02ce4553950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:34.018642 11229 raft_consensus.cc:1261] T 8b9436a5f7ea429e8d1700c0de0dabc5 P 87bfe270c0d142f6accbcc6339d6588b [term 1 FOLLOWER]: Refusing update from remote peer cceac09224e44e929841f02ce4553950: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:34.019032 11563 consensus_queue.cc:1035] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e47f2ce7ca58405a8d0afcfaec578182" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 35937 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:34.019204 11567 consensus_queue.cc:1035] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [LEADER]: Connected to new peer: Peer: permanent_uuid: "87bfe270c0d142f6accbcc6339d6588b" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 45695 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:34.022061 11571 mvcc.cc:205] Tried to move back new op lower bound from 7333484683337584640 to 7333484683161473024. Current Snapshot: MvccSnapshot[applied={T|T < 7333484683337584640}]
I20260926 05:45:34.021806 11570 mvcc.cc:205] Tried to move back new op lower bound from 7333484683337584640 to 7333484683161473024. Current Snapshot: MvccSnapshot[applied={T|T < 7333484683337584640}]
I20260926 05:45:34.024395 11083 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:34.024492 11083 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:32790:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:34.025197 11083 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1b81cd432805410b91ed3d3344afc9f3: Sending DeleteTablet for 3 replicas of tablet 8b9436a5f7ea429e8d1700c0de0dabc5
I20260926 05:45:34.025507 11343 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9436a5f7ea429e8d1700c0de0dabc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:34 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:38034
I20260926 05:45:34.025604 11477 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9436a5f7ea429e8d1700c0de0dabc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:34 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37308
I20260926 05:45:34.025705 11580 tablet_replica.cc:330] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: stopping tablet replica
I20260926 05:45:34.025815 11580 raft_consensus.cc:2229] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:34.025817 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11146
I20260926 05:45:34.025841 11209 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b9436a5f7ea429e8d1700c0de0dabc5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:34 UTC) from {username='kudu', principal='kudu/127.23.213.254@KRBTEST.COM'} at 127.0.0.1:37372
I20260926 05:45:34.025915 11580 raft_consensus.cc:2258] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:34.026563 11581 tablet_replica.cc:330] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: stopping tablet replica
I20260926 05:45:34.026662 11581 raft_consensus.cc:2229] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:45:34.026724 11580 ts_tablet_manager.cc:1905] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:34.026782 11581 raft_consensus.cc:2258] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:34.027205 11581 ts_tablet_manager.cc:1905] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:34.028515 11580 ts_tablet_manager.cc:1918] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:34.028599 11580 log.cc:1196] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/8b9436a5f7ea429e8d1700c0de0dabc5
I20260926 05:45:34.028878 11580 ts_tablet_manager.cc:1939] T 8b9436a5f7ea429e8d1700c0de0dabc5 P e47f2ce7ca58405a8d0afcfaec578182: Deleting consensus metadata
I20260926 05:45:34.029186 11581 ts_tablet_manager.cc:1918] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:34.029260 11581 log.cc:1196] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/8b9436a5f7ea429e8d1700c0de0dabc5
I20260926 05:45:34.029273 11068 catalog_manager.cc:4841] TS e47f2ce7ca58405a8d0afcfaec578182 (127.23.213.194:35937): tablet 8b9436a5f7ea429e8d1700c0de0dabc5 (table test-table [id=c8b2b15bba104b168aca3a95ee1da81d]) successfully deleted
I20260926 05:45:34.029533 11581 ts_tablet_manager.cc:1939] T 8b9436a5f7ea429e8d1700c0de0dabc5 P cceac09224e44e929841f02ce4553950: Deleting consensus metadata
I20260926 05:45:34.029955 11069 catalog_manager.cc:4841] TS cceac09224e44e929841f02ce4553950 (127.23.213.195:42761): tablet 8b9436a5f7ea429e8d1700c0de0dabc5 (table test-table [id=c8b2b15bba104b168aca3a95ee1da81d]) successfully deleted
W20260926 05:45:34.032879 11069 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:45695 (error 108)
I20260926 05:45:34.033329 11554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.033215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:45695 (local address 127.0.0.1:37378)
0926 05:45:34.033267 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.033301 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:45695: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 05:45:34.033476 11069 catalog_manager.cc:4584] TS 87bfe270c0d142f6accbcc6339d6588b (127.23.213.193:45695): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8b9436a5f7ea429e8d1700c0de0dabc5: Network error: Client connection negotiation failed: client connection to 127.23.213.193:45695: connect: Connection refused (error 111)
I20260926 05:45:34.033878 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11280
I20260926 05:45:34.039858 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11414
I20260926 05:45:34.044807 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11052
2026-09-26T05:45:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2108 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T05:45:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:34Z Disabled control of system clock
I20260926 05:45:34.072499 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:37947
--webserver_interface=127.23.213.254
--webserver_port=0
--builtin_ntp_servers=127.23.213.212:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:37947
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:34.153020 11586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:34.153231 11586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:34.153251 11586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:34.154358 11586 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:34.154395 11586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:34.154408 11586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:34.154421 11586 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:34.154433 11586 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:34.155695 11586 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:37947
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:37947
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:34.155882 11586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:34.156100 11586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:34.162266 11591 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:34.162266 11592 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:34.162685 11594 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:34.163034 11586 server_base.cc:1034] running on GCE node
I20260926 05:45:34.163328 11586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:34.163697 11586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:34.164889 11586 hybrid_clock.cc:648] HybridClock initialized: now 1790401534164868 us; error 35 us; skew 500 ppm
I20260926 05:45:34.166330 11586 webserver.cc:460] Webserver started at http://127.23.213.254:37431/ using document root <none> and password file <none>
I20260926 05:45:34.166587 11586 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:34.166631 11586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:34.166721 11586 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:34.167760 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "e32d2bf4a8664685ac4e01e14baaa334"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "a5408642af07c91b9eabe837b44d8b32"
server_key_iv: "c39f103f10a995ac875371ec6b720128"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.168058 11586 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "e32d2bf4a8664685ac4e01e14baaa334"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "a5408642af07c91b9eabe837b44d8b32"
server_key_iv: "c39f103f10a995ac875371ec6b720128"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.169324 11586 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:34.170065 11600 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.170284 11586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:34.170380 11586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "e32d2bf4a8664685ac4e01e14baaa334"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "a5408642af07c91b9eabe837b44d8b32"
server_key_iv: "c39f103f10a995ac875371ec6b720128"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.170440 11586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:34.204334 11586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:34.204646 11586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:34.204764 11586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:34.208726 11586 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:37947
I20260926 05:45:34.208796 11652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:37947 every 8 connection(s)
I20260926 05:45:34.209156 11586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:34.210191 11653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:34.212440 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334: Bootstrap starting.
I20260926 05:45:34.212953 11653 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:34.213183 11653 log.cc:824] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:34.213886 11653 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334: No bootstrap required, opened a new log
I20260926 05:45:34.215410 11653 raft_consensus.cc:348] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32d2bf4a8664685ac4e01e14baaa334" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37947 } }
I20260926 05:45:34.215546 11653 raft_consensus.cc:374] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:34.215569 11653 raft_consensus.cc:729] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e32d2bf4a8664685ac4e01e14baaa334, State: Initialized, Role: FOLLOWER
I20260926 05:45:34.215673 11653 consensus_queue.cc:260] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32d2bf4a8664685ac4e01e14baaa334" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37947 } }
I20260926 05:45:34.215757 11653 raft_consensus.cc:388] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:34.215795 11653 raft_consensus.cc:482] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:34.215868 11653 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:34.216403 11653 raft_consensus.cc:504] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32d2bf4a8664685ac4e01e14baaa334" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37947 } }
I20260926 05:45:34.216519 11653 leader_election.cc:302] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e32d2bf4a8664685ac4e01e14baaa334; no voters:
I20260926 05:45:34.216691 11653 leader_election.cc:288] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:34.216751 11656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:34.216926 11656 raft_consensus.cc:686] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [term 1 LEADER]: Becoming Leader. State: Replica: e32d2bf4a8664685ac4e01e14baaa334, State: Running, Role: LEADER
I20260926 05:45:34.216946 11653 sys_catalog.cc:564] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:34.217032 11656 consensus_queue.cc:237] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32d2bf4a8664685ac4e01e14baaa334" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37947 } }
I20260926 05:45:34.217285 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11586
I20260926 05:45:34.217393 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:34.217612 24407 external_mini_cluster.cc:1428] Setting key 8f6aac68852de331b481c21d9e67a118
I20260926 05:45:34.217864 11658 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e32d2bf4a8664685ac4e01e14baaa334. Latest consensus state: current_term: 1 leader_uuid: "e32d2bf4a8664685ac4e01e14baaa334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32d2bf4a8664685ac4e01e14baaa334" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37947 } } }
I20260926 05:45:34.218077 11658 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:34.218108 11657 sys_catalog.cc:455] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e32d2bf4a8664685ac4e01e14baaa334" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e32d2bf4a8664685ac4e01e14baaa334" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 37947 } } }
I20260926 05:45:34.218220 11657 sys_catalog.cc:458] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:34.218485 11667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:34.219096 11667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:34.221771 11667 catalog_manager.cc:1347] Generated new cluster ID: 94b211d56bc842d0a7081fe0131a2786
I20260926 05:45:34.221828 11667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:34.223781 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.218299 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56362 (local address 127.23.213.254:37947)
0926 05:45:34.218466 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.218475 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.218506 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:45:34.218671 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.218678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.218788 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.219249 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:34.219352 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.220584 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.221685 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.222197 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.222337 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.223030 (+ 693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:34.223034 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:34.223035 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:34.223240 (+ 205us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:34.223314 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:34.223329 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:34.223409 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.223492 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.223574 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":63,"threads_started":1}
I20260926 05:45:34.233922 11667 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:34.234620 11667 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:34.238731 11667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334: Generated new TSK 0
I20260926 05:45:34.238898 11667 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:34.241057 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.193:0
--local_ip_for_outbound_sockets=127.23.213.193
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:37947
--builtin_ntp_servers=127.23.213.212:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:34.330363 11677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:34.330598 11677 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:34.330633 11677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:34.331799 11677 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:34.331864 11677 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:34.331908 11677 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.193
I20260926 05:45:34.333273 11677 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.23.213.193
--webserver_port=0
--tserver_master_addrs=127.23.213.254:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11677
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.193
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:34.333540 11677 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:34.333880 11677 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:34.339813 11682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:34.339826 11683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:34.339802 11685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:34.340175 11677 server_base.cc:1034] running on GCE node
I20260926 05:45:34.340406 11677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:34.340695 11677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:34.341830 11677 hybrid_clock.cc:648] HybridClock initialized: now 1790401534341817 us; error 26 us; skew 500 ppm
I20260926 05:45:34.343420 11677 webserver.cc:460] Webserver started at http://127.23.213.193:35797/ using document root <none> and password file <none>
I20260926 05:45:34.343719 11677 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:34.343788 11677 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:34.343926 11677 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:34.345444 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data/instance:
uuid: "c6dfd62a6ca541e0b46cc2560e355da9"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "6bd7a5b5759c7fce5a3b68dc6d678b5b"
server_key_iv: "7eed9f691fb54f8b1fdeec198505405b"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.345884 11677 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance:
uuid: "c6dfd62a6ca541e0b46cc2560e355da9"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "6bd7a5b5759c7fce5a3b68dc6d678b5b"
server_key_iv: "7eed9f691fb54f8b1fdeec198505405b"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.347608 11677 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 05:45:34.348516 11691 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.348701 11677 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:34.348796 11677 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/wal
uuid: "c6dfd62a6ca541e0b46cc2560e355da9"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "6bd7a5b5759c7fce5a3b68dc6d678b5b"
server_key_iv: "7eed9f691fb54f8b1fdeec198505405b"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.348870 11677 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:34.357966 11677 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:34.358325 11677 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:34.358464 11677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:34.358695 11677 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:34.359036 11677 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:34.359094 11677 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.359123 11677 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:34.359148 11677 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.366266 11677 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.193:42127
I20260926 05:45:34.366472 11804 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.193:42127 every 8 connection(s)
I20260926 05:45:34.366707 11677 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/data/info.pb
I20260926 05:45:34.372049 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.367435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.193:44693 (local address 127.23.213.254:37947)
0926 05:45:34.367478 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.367480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.367720 (+ 240us) server_negotiation.cc:407] Connection header received
0926 05:45:34.368246 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.368247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.368261 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.368312 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:34.369019 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.369543 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.370609 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.370824 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.370910 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.371766 (+ 856us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:34.371767 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:34.371767 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:34.371773 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:34.371789 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:34.371793 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:34.371847 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.371938 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.371970 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:34.372272 11807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.367433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:37947 (local address 127.23.213.193:44693)
0926 05:45:34.367604 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.367637 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.368138 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.368389 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.368393 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.368401 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:34.368927 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.368931 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.369637 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.369638 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.370517 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.370519 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.370922 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.370923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.371056 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.371607 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:34.371608 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:34.371636 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:34.371659 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:34.371849 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:34.371854 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:34.371857 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.371942 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.372041 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":83,"threads_started":1}
I20260926 05:45:34.372773 11805 heartbeater.cc:347] Connected to a master server at 127.23.213.254:37947
I20260926 05:45:34.372850 11805 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:34.373045 11805 heartbeater.cc:511] Master 127.23.213.254:37947 requested a full tablet report, sending...
I20260926 05:45:34.373553 11617 ts_manager.cc:194] Registered new tserver with Master: c6dfd62a6ca541e0b46cc2560e355da9 (127.23.213.193:42127)
I20260926 05:45:34.374219 11617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.193:44693
I20260926 05:45:34.376554 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11677
I20260926 05:45:34.376637 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-0/wal/instance
I20260926 05:45:34.376845 24407 external_mini_cluster.cc:1428] Setting key 41fd8f9f5fb655e4701142f6474da171
I20260926 05:45:34.377862 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.194:0
--local_ip_for_outbound_sockets=127.23.213.194
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:37947
--builtin_ntp_servers=127.23.213.212:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 05:45:34.457940 11808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:34.458138 11808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:34.458159 11808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:34.459298 11808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:34.459337 11808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:34.459362 11808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.194
I20260926 05:45:34.460664 11808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.23.213.194
--webserver_port=0
--tserver_master_addrs=127.23.213.254:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.194
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:34.460896 11808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:34.461182 11808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:34.466179 11813 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:34.466187 11814 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:34.466219 11808 server_base.cc:1034] running on GCE node
W20260926 05:45:34.466197 11816 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:34.466691 11808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:34.466902 11808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:34.468048 11808 hybrid_clock.cc:648] HybridClock initialized: now 1790401534468035 us; error 28 us; skew 500 ppm
I20260926 05:45:34.469249 11808 webserver.cc:460] Webserver started at http://127.23.213.194:39367/ using document root <none> and password file <none>
I20260926 05:45:34.469501 11808 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:34.469545 11808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:34.469691 11808 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:34.470798 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data/instance:
uuid: "932732a6b16e4da28dfc81ee44bc1487"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "15c62482056ab325aa82c875d2a6a435"
server_key_iv: "d45880b3ef9bd734e0544437355642bd"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.471150 11808 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance:
uuid: "932732a6b16e4da28dfc81ee44bc1487"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "15c62482056ab325aa82c875d2a6a435"
server_key_iv: "d45880b3ef9bd734e0544437355642bd"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.472451 11808 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:34.473261 11822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.473489 11808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:34.473584 11808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal
uuid: "932732a6b16e4da28dfc81ee44bc1487"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "15c62482056ab325aa82c875d2a6a435"
server_key_iv: "d45880b3ef9bd734e0544437355642bd"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.473671 11808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:34.491216 11808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:34.491484 11808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:34.491585 11808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:34.491762 11808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:34.492139 11808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:34.492214 11808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.492269 11808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:34.492298 11808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.497874 11808 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.194:33793
I20260926 05:45:34.497951 11935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.194:33793 every 8 connection(s)
I20260926 05:45:34.498291 11808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/data/info.pb
I20260926 05:45:34.502435 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11808
I20260926 05:45:34.502548 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal/instance
I20260926 05:45:34.502736 24407 external_mini_cluster.cc:1428] Setting key 3fec0ea82f40990f80a8e25ff88c8e1f
I20260926 05:45:34.503434 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.499028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.194:47967 (local address 127.23.213.254:37947)
0926 05:45:34.499100 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.499104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.499347 (+ 243us) server_negotiation.cc:407] Connection header received
0926 05:45:34.499812 (+ 465us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.499813 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.499824 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.499873 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:34.500505 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.500980 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.501986 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.502166 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.502219 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.503114 (+ 895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:34.503115 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:34.503116 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:34.503121 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:34.503137 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:34.503142 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:34.503205 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.503321 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.503344 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 05:45:34.503724 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.499025 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:37947 (local address 127.23.213.194:47967)
0926 05:45:34.499226 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.499260 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.499730 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.499934 (+ 204us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.499937 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.499944 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:34.500438 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.500443 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.501060 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.501060 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.501912 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.501914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.502236 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.502237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.502359 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.502961 (+ 602us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:34.502962 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:34.503001 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:34.503031 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:34.503227 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:34.503234 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:34.503237 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.503311 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.503418 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":94,"threads_started":1}
I20260926 05:45:34.503887 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.23.213.195:0
--local_ip_for_outbound_sockets=127.23.213.195
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:37947
--builtin_ntp_servers=127.23.213.212:35265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 05:45:34.504298 11936 heartbeater.cc:347] Connected to a master server at 127.23.213.254:37947
I20260926 05:45:34.504411 11936 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:34.504587 11936 heartbeater.cc:511] Master 127.23.213.254:37947 requested a full tablet report, sending...
I20260926 05:45:34.505028 11617 ts_manager.cc:194] Registered new tserver with Master: 932732a6b16e4da28dfc81ee44bc1487 (127.23.213.194:33793)
I20260926 05:45:34.505473 11617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.194:47967
W20260926 05:45:34.585736 11939 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:34.585949 11939 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:34.585971 11939 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:34.587139 11939 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:34.587184 11939 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:34.587210 11939 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.23.213.195
I20260926 05:45:34.588584 11939 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:35265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.23.213.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.23.213.195
--webserver_port=0
--tserver_master_addrs=127.23.213.254:37947
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11939
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.23.213.195
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:34.588846 11939 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:34.589155 11939 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:34.594156 11947 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:34.594151 11944 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:34.594194 11945 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:34.594208 11939 server_base.cc:1034] running on GCE node
I20260926 05:45:34.594630 11939 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:34.594887 11939 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:34.596050 11939 hybrid_clock.cc:648] HybridClock initialized: now 1790401534596041 us; error 31 us; skew 500 ppm
I20260926 05:45:34.597256 11939 webserver.cc:460] Webserver started at http://127.23.213.195:33085/ using document root <none> and password file <none>
I20260926 05:45:34.597512 11939 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:34.597582 11939 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:34.597733 11939 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:34.598850 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data/instance:
uuid: "af60cc17bd87405b8e568178e214238a"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "e910feba7a43092ac3ae4a46c27bf2a2"
server_key_iv: "818f2c825bd74186a0b9524a116e3c42"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.599205 11939 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance:
uuid: "af60cc17bd87405b8e568178e214238a"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "e910feba7a43092ac3ae4a46c27bf2a2"
server_key_iv: "818f2c825bd74186a0b9524a116e3c42"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.600442 11939 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:45:34.601104 11953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.601270 11939 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:45:34.601365 11939 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal
uuid: "af60cc17bd87405b8e568178e214238a"
format_stamp: "Formatted at 2026-09-26 05:45:34 on dist-test-slave-1fdt"
server_key: "e910feba7a43092ac3ae4a46c27bf2a2"
server_key_iv: "818f2c825bd74186a0b9524a116e3c42"
server_key_version: "encryptionkey@0"
I20260926 05:45:34.601447 11939 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:34.610800 11939 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:34.611081 11939 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:34.611225 11939 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:34.611433 11939 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 05:45:34.611748 11939 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 05:45:34.611804 11939 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.611845 11939 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 05:45:34.611872 11939 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:34.617612 11939 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.195:45837
I20260926 05:45:34.617697 12066 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.195:45837 every 8 connection(s)
I20260926 05:45:34.618053 11939 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/data/info.pb
I20260926 05:45:34.619922 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 11939
I20260926 05:45:34.620039 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal/instance
I20260926 05:45:34.620239 24407 external_mini_cluster.cc:1428] Setting key c33ad49050692300e984606ce851d888
I20260926 05:45:34.623481 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.618776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:50187 (local address 127.23.213.254:37947)
0926 05:45:34.618846 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.618848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.619086 (+ 238us) server_negotiation.cc:407] Connection header received
0926 05:45:34.619535 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.619536 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.619546 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.619596 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:34.620312 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.620913 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.622019 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.622217 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.622303 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.623152 (+ 849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:34.623153 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:34.623154 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:34.623166 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:34.623181 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:34.623186 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:34.623248 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.623370 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.623401 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:45:34.623718 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.618801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:37947 (local address 127.23.213.195:50187)
0926 05:45:34.618966 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.618999 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.619456 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.619646 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.619649 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.619656 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:34.620214 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.620219 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.621007 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.621008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.621925 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.621927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.622313 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.622314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.622455 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.623006 (+ 551us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 05:45:34.623007 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:34.623034 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 05:45:34.623056 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:34.623263 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:34.623269 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:34.623272 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.623372 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.623474 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":82,"threads_started":1}
I20260926 05:45:34.624329 12067 heartbeater.cc:347] Connected to a master server at 127.23.213.254:37947
I20260926 05:45:34.624421 12067 heartbeater.cc:464] Registering TS with master...
I20260926 05:45:34.624571 12067 heartbeater.cc:511] Master 127.23.213.254:37947 requested a full tablet report, sending...
I20260926 05:45:34.624943 11617 ts_manager.cc:194] Registered new tserver with Master: af60cc17bd87405b8e568178e214238a (127.23.213.195:45837)
I20260926 05:45:34.625300 11617 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.23.213.195:50187
I20260926 05:45:34.632197 24407 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 05:45:34.636555 11668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.633618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56374 (local address 127.23.213.254:37947)
0926 05:45:34.633696 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.633698 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.633803 (+ 105us) server_negotiation.cc:407] Connection header received
0926 05:45:34.633873 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.633874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.633886 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.633957 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:34.634123 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.634593 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.635349 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.635539 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.635624 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.636312 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:34.636314 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:34.636314 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:34.636319 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:34.636333 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:34.636337 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:34.636390 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.636448 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.636473 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 05:45:34.638150 11617 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56374:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 05:45:34.639706 11617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 05:45:34.645726 12075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.641806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.195:45837 (local address 127.0.0.1:48778)
0926 05:45:34.641926 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.641955 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.642033 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.642226 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.642230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.642231 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:34.642288 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.642290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.643500 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.643501 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.644500 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.644502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.645406 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.645407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.645481 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.645493 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.645603 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.645655 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260926 05:45:34.645764 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.641884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48778 (local address 127.23.213.195:45837)
0926 05:45:34.642055 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.642059 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.642070 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:45:34.642107 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.642109 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.642124 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.642211 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:34.642556 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.643412 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.644594 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.645323 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.645406 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.645496 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.645619 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.645657 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":90,"threads_started":1}
I20260926 05:45:34.646234 12078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.642163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:33793 (local address 127.0.0.1:51148)
0926 05:45:34.642427 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.642437 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.642474 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.642759 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.642760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.642762 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:34.642833 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.642835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.644079 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.644080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.645133 (+ 1053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.645135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.646013 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.646014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.646073 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.646076 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.646132 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.646166 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":246,"thread_start_us":54,"threads_started":1}
I20260926 05:45:34.646281 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.642341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51148 (local address 127.23.213.194:33793)
0926 05:45:34.642579 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.642584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.642594 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:45:34.642639 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.642641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.642656 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.642765 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:34.642893 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.643978 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.645220 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.645925 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.646005 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.646093 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.646149 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.646196 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":115,"threads_started":1}
I20260926 05:45:34.646551 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.642001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:42127 (local address 127.0.0.1:48190)
0926 05:45:34.642115 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.642124 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.642168 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.642858 (+ 690us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.642859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.642862 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:34.642904 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.642905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.644109 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.644110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.644996 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.644999 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.646177 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.646178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.646241 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.646245 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.646440 (+ 195us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.646481 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":66,"threads_started":1}
I20260926 05:45:34.646673 12001 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0cd54cb3b84760a632964793b38825 (DEFAULT_TABLE table=test-table [id=42a2b84ffba44e7d9c8e1d6cb09a061e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:34.646971 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.642401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48190 (local address 127.23.213.193:42127)
0926 05:45:34.642608 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.642613 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.642623 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:45:34.642661 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.642664 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.642678 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.642767 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:34.642967 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.644026 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.645082 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.646009 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.646068 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.646140 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.646851 (+ 711us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.646897 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":89,"threads_started":1}
I20260926 05:45:34.646898 11870 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0cd54cb3b84760a632964793b38825 (DEFAULT_TABLE table=test-table [id=42a2b84ffba44e7d9c8e1d6cb09a061e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:34.647100 12001 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0cd54cb3b84760a632964793b38825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:34.647203 11870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0cd54cb3b84760a632964793b38825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:34.649534 12081 tablet_bootstrap.cc:492] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Bootstrap starting.
I20260926 05:45:34.649883 12082 tablet_bootstrap.cc:492] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Bootstrap starting.
I20260926 05:45:34.650103 11739 tablet_service.cc:1467] Processing CreateTablet for tablet 0a0cd54cb3b84760a632964793b38825 (DEFAULT_TABLE table=test-table [id=42a2b84ffba44e7d9c8e1d6cb09a061e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 05:45:34.650230 12081 tablet_bootstrap.cc:654] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:34.650382 11739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a0cd54cb3b84760a632964793b38825. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:34.650492 12082 tablet_bootstrap.cc:654] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:34.650528 12081 log.cc:824] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:34.650820 12082 log.cc:824] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:34.651294 12081 tablet_bootstrap.cc:492] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: No bootstrap required, opened a new log
I20260926 05:45:34.651353 12081 ts_tablet_manager.cc:1397] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:34.651424 12082 tablet_bootstrap.cc:492] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: No bootstrap required, opened a new log
I20260926 05:45:34.651464 12082 ts_tablet_manager.cc:1397] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:34.653645 12082 raft_consensus.cc:348] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.653847 12082 raft_consensus.cc:374] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:34.653898 12082 raft_consensus.cc:729] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 932732a6b16e4da28dfc81ee44bc1487, State: Initialized, Role: FOLLOWER
I20260926 05:45:34.653936 12085 tablet_bootstrap.cc:492] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: Bootstrap starting.
I20260926 05:45:34.653928 12081 raft_consensus.cc:348] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.654035 12082 consensus_queue.cc:260] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.654091 12081 raft_consensus.cc:374] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:34.654131 12081 raft_consensus.cc:729] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af60cc17bd87405b8e568178e214238a, State: Initialized, Role: FOLLOWER
I20260926 05:45:34.654247 12081 consensus_queue.cc:260] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.654340 12082 ts_tablet_manager.cc:1428] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:45:34.654460 11936 heartbeater.cc:503] Master 127.23.213.254:37947 was elected leader, sending a full tablet report...
I20260926 05:45:34.654521 12081 ts_tablet_manager.cc:1428] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 05:45:34.654570 12067 heartbeater.cc:503] Master 127.23.213.254:37947 was elected leader, sending a full tablet report...
I20260926 05:45:34.654580 12085 tablet_bootstrap.cc:654] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:34.654891 12085 log.cc:824] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:34.655799 12085 tablet_bootstrap.cc:492] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: No bootstrap required, opened a new log
I20260926 05:45:34.655874 12085 ts_tablet_manager.cc:1397] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 05:45:34.657728 12085 raft_consensus.cc:348] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.657915 12085 raft_consensus.cc:374] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:34.657939 12085 raft_consensus.cc:729] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6dfd62a6ca541e0b46cc2560e355da9, State: Initialized, Role: FOLLOWER
I20260926 05:45:34.658046 12085 consensus_queue.cc:260] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.658298 12085 ts_tablet_manager.cc:1428] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 05:45:34.658370 11805 heartbeater.cc:503] Master 127.23.213.254:37947 was elected leader, sending a full tablet report...
W20260926 05:45:34.748859 11937 tablet.cc:2369] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:34.837101 12087 raft_consensus.cc:482] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 05:45:34.837276 12087 raft_consensus.cc:504] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.837685 12087 leader_election.cc:288] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6dfd62a6ca541e0b46cc2560e355da9 (127.23.213.193:42127), 932732a6b16e4da28dfc81ee44bc1487 (127.23.213.194:33793)
I20260926 05:45:34.841697 12080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.838006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:35905 (local address 127.23.213.193:42127)
0926 05:45:34.838069 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.838071 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.838084 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:45:34.838118 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.838119 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.838138 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.838240 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:34.838444 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.839208 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.840790 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.841334 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.841408 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.841416 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.841591 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.841628 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:45:34.841702 12079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.838147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.23.213.195:55359 (local address 127.23.213.194:33793)
0926 05:45:34.838211 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.838213 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.838258 (+ 45us) server_negotiation.cc:407] Connection header received
0926 05:45:34.838321 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.838322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.838337 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.838379 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 05:45:34.838539 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.839232 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.840780 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.841334 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.841408 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.841416 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.841591 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.841628 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:45:34.841712 12090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.838036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.194:33793 (local address 127.23.213.195:55359)
0926 05:45:34.838194 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.838203 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.838268 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.838407 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.838408 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.838410 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:34.838471 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.838472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.839304 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.839304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.840682 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.840685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.841423 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.841424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.841506 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.841510 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.841565 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.841614 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":98,"thread_start_us":98,"threads_started":1}
I20260926 05:45:34.841706 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.837908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:42127 (local address 127.23.213.195:35905)
0926 05:45:34.837972 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.837985 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:34.838032 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:34.838258 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:34.838260 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:34.838261 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 05:45:34.838348 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.838351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.839304 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.839304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.840682 (+ 1378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:34.840685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.841422 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.841423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:34.841505 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.841510 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 05:45:34.841565 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 05:45:34.841614 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":124}
I20260926 05:45:34.842087 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0cd54cb3b84760a632964793b38825" candidate_uuid: "af60cc17bd87405b8e568178e214238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932732a6b16e4da28dfc81ee44bc1487" is_pre_election: true
I20260926 05:45:34.842293 11890 raft_consensus.cc:2454] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af60cc17bd87405b8e568178e214238a in term 0.
I20260926 05:45:34.842315 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0cd54cb3b84760a632964793b38825" candidate_uuid: "af60cc17bd87405b8e568178e214238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" is_pre_election: true
I20260926 05:45:34.842485 11759 raft_consensus.cc:2454] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate af60cc17bd87405b8e568178e214238a in term 0.
I20260926 05:45:34.842554 11956 leader_election.cc:302] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 932732a6b16e4da28dfc81ee44bc1487, af60cc17bd87405b8e568178e214238a; no voters:
I20260926 05:45:34.842702 12087 raft_consensus.cc:2781] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 05:45:34.842804 12087 raft_consensus.cc:482] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 05:45:34.842844 12087 raft_consensus.cc:3037] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:34.843538 12087 raft_consensus.cc:504] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.843683 12087 leader_election.cc:288] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [CANDIDATE]: Term 1 election: Requested vote from peers c6dfd62a6ca541e0b46cc2560e355da9 (127.23.213.193:42127), 932732a6b16e4da28dfc81ee44bc1487 (127.23.213.194:33793)
I20260926 05:45:34.843865 11759 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0cd54cb3b84760a632964793b38825" candidate_uuid: "af60cc17bd87405b8e568178e214238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6dfd62a6ca541e0b46cc2560e355da9"
I20260926 05:45:34.843865 11890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a0cd54cb3b84760a632964793b38825" candidate_uuid: "af60cc17bd87405b8e568178e214238a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "932732a6b16e4da28dfc81ee44bc1487"
I20260926 05:45:34.843936 11759 raft_consensus.cc:3037] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:34.843961 11890 raft_consensus.cc:3037] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:34.844671 11759 raft_consensus.cc:2454] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af60cc17bd87405b8e568178e214238a in term 1.
I20260926 05:45:34.844681 11890 raft_consensus.cc:2454] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate af60cc17bd87405b8e568178e214238a in term 1.
I20260926 05:45:34.844839 11956 leader_election.cc:302] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 932732a6b16e4da28dfc81ee44bc1487, af60cc17bd87405b8e568178e214238a; no voters:
I20260926 05:45:34.844949 12087 raft_consensus.cc:2781] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:34.845112 12087 raft_consensus.cc:686] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 1 LEADER]: Becoming Leader. State: Replica: af60cc17bd87405b8e568178e214238a, State: Running, Role: LEADER
I20260926 05:45:34.845207 12087 consensus_queue.cc:237] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } }
I20260926 05:45:34.846064 11616 catalog_manager.cc:5495] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a reported cstate change: term changed from 0 to 1, leader changed from <none> to af60cc17bd87405b8e568178e214238a (127.23.213.195). New cstate: current_term: 1 leader_uuid: "af60cc17bd87405b8e568178e214238a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "af60cc17bd87405b8e568178e214238a" member_type: VOTER last_known_addr { host: "127.23.213.195" port: 45837 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 } health_report { overall_health: UNKNOWN } } }
W20260926 05:45:34.867487 11806 tablet.cc:2369] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 05:45:34.868660 12068 tablet.cc:2369] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 05:45:34.871059 12076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.868839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48788 (local address 127.23.213.195:45837)
0926 05:45:34.868889 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:34.868891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:34.868904 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:45:34.869014 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:34.869015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:34.869031 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:34.869090 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:45:34.869216 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.869665 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.870454 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:34.870641 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:34.870693 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:34.870823 (+ 130us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:45:34.870916 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:45:34.870975 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:34.870993 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 05:45:34.871009 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:34.872841 11759 raft_consensus.cc:1261] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9 [term 1 FOLLOWER]: Refusing update from remote peer af60cc17bd87405b8e568178e214238a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:34.872901 11890 raft_consensus.cc:1261] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 1 FOLLOWER]: Refusing update from remote peer af60cc17bd87405b8e568178e214238a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:45:34.873184 12091 consensus_queue.cc:1035] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6dfd62a6ca541e0b46cc2560e355da9" member_type: VOTER last_known_addr { host: "127.23.213.193" port: 42127 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:34.873349 12087 consensus_queue.cc:1035] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [LEADER]: Connected to new peer: Peer: permanent_uuid: "932732a6b16e4da28dfc81ee44bc1487" member_type: VOTER last_known_addr { host: "127.23.213.194" port: 33793 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:45:34.875887 12095 mvcc.cc:205] Tried to move back new op lower bound from 7333484686836346880 to 7333484686726623232. Current Snapshot: MvccSnapshot[applied={T|T < 7333484686836346880}]
I20260926 05:45:34.878037 11617 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:34.878132 11617 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56374:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 05:45:34.878901 11617 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e32d2bf4a8664685ac4e01e14baaa334: Sending DeleteTablet for 3 replicas of tablet 0a0cd54cb3b84760a632964793b38825
I20260926 05:45:34.879215 12001 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0cd54cb3b84760a632964793b38825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:34 UTC) from {username='slave'} at 127.0.0.1:48778
I20260926 05:45:34.879258 11739 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0cd54cb3b84760a632964793b38825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:34 UTC) from {username='slave'} at 127.0.0.1:48190
I20260926 05:45:34.879317 11870 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a0cd54cb3b84760a632964793b38825 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 05:45:34 UTC) from {username='slave'} at 127.0.0.1:51148
I20260926 05:45:34.879541 12104 tablet_replica.cc:330] T 0a0cd54cb3b84760a632964793b38825 P c6dfd62a6ca541e0b46cc2560e355da9: stopping tablet replica
I20260926 05:45:34.879602 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11677
I20260926 05:45:34.881211 12103 tablet_replica.cc:330] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: stopping tablet replica
I20260926 05:45:34.881322 12103 raft_consensus.cc:2229] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 1 LEADER]: Raft consensus shutting down.
I20260926 05:45:34.881410 12103 raft_consensus.cc:2258] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:34.881670 12105 tablet_replica.cc:330] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: stopping tablet replica
I20260926 05:45:34.881781 12105 raft_consensus.cc:2229] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 05:45:34.881879 12105 raft_consensus.cc:2258] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 05:45:34.881932 12103 ts_tablet_manager.cc:1905] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:34.882269 12105 ts_tablet_manager.cc:1905] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 05:45:34.883642 12103 ts_tablet_manager.cc:1918] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:34.883719 12103 log.cc:1196] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-2/wal/wals/0a0cd54cb3b84760a632964793b38825
I20260926 05:45:34.883982 12103 ts_tablet_manager.cc:1939] T 0a0cd54cb3b84760a632964793b38825 P af60cc17bd87405b8e568178e214238a: Deleting consensus metadata
I20260926 05:45:34.884094 12105 ts_tablet_manager.cc:1918] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 05:45:34.884155 12105 log.cc:1196] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Deleting WAL directory at /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790401292175461-24407-0/minicluster-data/ts-1/wal/wals/0a0cd54cb3b84760a632964793b38825
I20260926 05:45:34.884387 12105 ts_tablet_manager.cc:1939] T 0a0cd54cb3b84760a632964793b38825 P 932732a6b16e4da28dfc81ee44bc1487: Deleting consensus metadata
I20260926 05:45:34.884516 11603 catalog_manager.cc:4841] TS af60cc17bd87405b8e568178e214238a (127.23.213.195:45837): tablet 0a0cd54cb3b84760a632964793b38825 (table test-table [id=42a2b84ffba44e7d9c8e1d6cb09a061e]) successfully deleted
I20260926 05:45:34.884743 11603 catalog_manager.cc:4841] TS 932732a6b16e4da28dfc81ee44bc1487 (127.23.213.194:33793): tablet 0a0cd54cb3b84760a632964793b38825 (table test-table [id=42a2b84ffba44e7d9c8e1d6cb09a061e]) successfully deleted
W20260926 05:45:34.886405 11603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.23.213.193:42127 (error 108)
I20260926 05:45:34.886713 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11808
I20260926 05:45:34.887040 12077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:34.886921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.193:42127 (local address 127.0.0.1:48200)
0926 05:45:34.886969 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:34.887001 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.193:42127: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 05:45:34.888474 11603 catalog_manager.cc:4584] TS c6dfd62a6ca541e0b46cc2560e355da9 (127.23.213.193:42127): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 0a0cd54cb3b84760a632964793b38825: Network error: Client connection negotiation failed: client connection to 127.23.213.193:42127: connect: Connection refused (error 111)
I20260926 05:45:34.891659 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11939
I20260926 05:45:34.896189 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 11586
2026-09-26T05:45:34Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (849 ms)
[----------] 6 tests from EncryptionPolicyTest (9009 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:34 dist-test-slave-1fdt krb5kdc[12109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:34 dist-test-slave-1fdt krb5kdc[12109](info): set up 2 sockets
Sep 26 05:45:34 dist-test-slave-1fdt krb5kdc[12109](info): commencing operation
krb5kdc: starting...
W20260926 05:45:35.933667 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:36 dist-test-slave-1fdt krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:36.052934 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38489
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33665
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38489
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:36.132771 12122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:36.132974 12122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:36.132997 12122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:36.134104 12122 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:36.134140 12122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:36.134155 12122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:36.134167 12122 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:36.134181 12122 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:36.135584 12122 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33665
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38489
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:36.135790 12122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:36.135964 12122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:36.140779 12130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:36.140777 12127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:36.140777 12128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:36.141032 12122 server_base.cc:1034] running on GCE node
I20260926 05:45:36.141273 12122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:36.141601 12122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:36.142792 12122 hybrid_clock.cc:648] HybridClock initialized: now 1790401536142766 us; error 39 us; skew 500 ppm
Sep 26 05:45:36 dist-test-slave-1fdt krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401536, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:36.149466 12122 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:36.150013 12122 webserver.cc:460] Webserver started at http://127.0.0.1:46339/ using document root <none> and password file <none>
I20260926 05:45:36.150262 12122 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:36.150332 12122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:36.150478 12122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:36.151556 12122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "8c932bdf76604e5890f015c144d76d80"
format_stamp: "Formatted at 2026-09-26 05:45:36 on dist-test-slave-1fdt"
server_key: "93c4a2e6810ef795a445cdb113c03908"
server_key_iv: "82fc024205d3da41665fb3044b0dea35"
server_key_version: "encryptionkey@0"
I20260926 05:45:36.151928 12122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "8c932bdf76604e5890f015c144d76d80"
format_stamp: "Formatted at 2026-09-26 05:45:36 on dist-test-slave-1fdt"
server_key: "93c4a2e6810ef795a445cdb113c03908"
server_key_iv: "82fc024205d3da41665fb3044b0dea35"
server_key_version: "encryptionkey@0"
I20260926 05:45:36.153234 12122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:36.154037 12137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:36.154204 12122 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:36.154313 12122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "8c932bdf76604e5890f015c144d76d80"
format_stamp: "Formatted at 2026-09-26 05:45:36 on dist-test-slave-1fdt"
server_key: "93c4a2e6810ef795a445cdb113c03908"
server_key_iv: "82fc024205d3da41665fb3044b0dea35"
server_key_version: "encryptionkey@0"
I20260926 05:45:36.154389 12122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:36.169996 12122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:36.171844 12122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:36.172012 12122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:36.176189 12122 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38489
I20260926 05:45:36.176241 12189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38489 every 8 connection(s)
I20260926 05:45:36.176635 12122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:36.177508 12190 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:36.177780 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12122
I20260926 05:45:36.177901 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:36.178108 24407 external_mini_cluster.cc:1428] Setting key b9ee88ccab24ddbf8e6fe79b39ea1322
I20260926 05:45:36.180497 12190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80: Bootstrap starting.
I20260926 05:45:36.181113 12190 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:36.181401 12190 log.cc:824] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:36.182122 12190 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80: No bootstrap required, opened a new log
I20260926 05:45:36.184161 12190 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c932bdf76604e5890f015c144d76d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38489 } }
I20260926 05:45:36.184331 12190 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:36.184366 12190 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c932bdf76604e5890f015c144d76d80, State: Initialized, Role: FOLLOWER
I20260926 05:45:36.184521 12190 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c932bdf76604e5890f015c144d76d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38489 } }
I20260926 05:45:36.184599 12190 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:36.184629 12190 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:36.184674 12190 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 05:45:36 dist-test-slave-1fdt krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401536, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:45:36.185271 12190 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c932bdf76604e5890f015c144d76d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38489 } }
I20260926 05:45:36.185359 12190 leader_election.cc:302] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c932bdf76604e5890f015c144d76d80; no voters:
I20260926 05:45:36.185539 12190 leader_election.cc:288] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:36.185570 12195 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:36.185685 12195 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [term 1 LEADER]: Becoming Leader. State: Replica: 8c932bdf76604e5890f015c144d76d80, State: Running, Role: LEADER
I20260926 05:45:36.185786 12195 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c932bdf76604e5890f015c144d76d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38489 } }
I20260926 05:45:36.185817 12190 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:36.186285 12195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c932bdf76604e5890f015c144d76d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c932bdf76604e5890f015c144d76d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38489 } } }
I20260926 05:45:36.186379 12195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:36.186486 12195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c932bdf76604e5890f015c144d76d80. Latest consensus state: current_term: 1 leader_uuid: "8c932bdf76604e5890f015c144d76d80" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c932bdf76604e5890f015c144d76d80" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38489 } } }
I20260926 05:45:36.186545 12195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80 [sys.catalog]: This master's current role is: LEADER
W20260926 05:45:36.187844 12210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:45:36.187889 12210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:45:36.187943 12201 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:36.188081 12201 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:36.189339 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:36.178654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32869 (local address 127.0.0.1:38489)
0926 05:45:36.178860 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:36.178869 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:36.178893 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:45:36.179288 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:36.179294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:36.179351 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:36.179463 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:36.180229 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:36.181191 (+ 962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:36.182120 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:36.182595 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:36.182722 (+ 127us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:36.185160 (+ 2438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:36.185164 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:36.185164 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:36.185175 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:36.187466 (+ 2291us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:36.188012 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:36.188016 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:36.188016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:36.188122 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:36.188388 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:36.188389 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:36.188390 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:36.188646 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:36.188796 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:36.189019 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 05:45:36.189127 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":93,"threads_started":1}
I20260926 05:45:36.189881 12201 catalog_manager.cc:1347] Generated new cluster ID: 0cb0750762f344bc97cbd5d780b8c20e
I20260926 05:45:36.189929 12201 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:36.190325 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:36.197947 12193 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:36.191610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35359 (local address 127.0.0.1:38489)
0926 05:45:36.191646 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:36.191647 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:36.191768 (+ 121us) server_negotiation.cc:407] Connection header received
0926 05:45:36.191839 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:36.191840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:36.191849 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:36.191897 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:36.192382 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:36.193110 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:36.193850 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:36.194080 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:36.194158 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:36.195311 (+ 1153us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:36.195312 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:36.195312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:36.195319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:36.196719 (+ 1400us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:36.197103 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:36.197104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:36.197104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:36.197165 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:36.197392 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:36.197393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:36.197394 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:36.197506 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:36.197616 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:36.197765 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 05:45:36.197830 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:36.199453 12201 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:36.199929 12201 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:36.205013 12201 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c932bdf76604e5890f015c144d76d80: Generated new TSK 0
I20260926 05:45:36.205191 12201 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:36.211066 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12122
W20260926 05:45:36.215816 12107 connection.cc:535] client connection to 127.0.0.1:38489 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38489): Connection reset by peer (error 104)
2026-09-26T05:45:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1314 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:36 dist-test-slave-1fdt krb5kdc[12221](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:36 dist-test-slave-1fdt krb5kdc[12221](info): set up 2 sockets
Sep 26 05:45:36 dist-test-slave-1fdt krb5kdc[12221](info): commencing operation
krb5kdc: starting...
W20260926 05:45:37.241890 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.012s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:37 dist-test-slave-1fdt krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401537, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:37.358862 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38107
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37169
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38107
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:37.437973 12234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:37.438210 12234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:37.438234 12234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:37.439325 12234 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:37.439363 12234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:37.439378 12234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:37.439389 12234 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:37.439401 12234 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:37.440799 12234 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37169
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38107
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12234
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:37.440999 12234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:37.441192 12234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:45:37.446059 12234 server_base.cc:1034] running on GCE node
W20260926 05:45:37.446012 12239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:37.446012 12242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:37.446027 12240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:37.446557 12234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:37.446859 12234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:37.448028 12234 hybrid_clock.cc:648] HybridClock initialized: now 1790401537448023 us; error 34 us; skew 500 ppm
Sep 26 05:45:37 dist-test-slave-1fdt krb5kdc[12221](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401537, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:37.450431 12234 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:37.450868 12234 webserver.cc:460] Webserver started at http://127.0.0.1:33077/ using document root <none> and password file <none>
I20260926 05:45:37.451108 12234 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:37.451175 12234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:37.451331 12234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:37.452382 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "5f25989b8cf6478ca806176020ba426a"
format_stamp: "Formatted at 2026-09-26 05:45:37 on dist-test-slave-1fdt"
server_key: "b2ed58f82f701466d6ff3925efdcaf1b"
server_key_iv: "25e14127acb1f98c5a4742a3fd0387bf"
server_key_version: "encryptionkey@0"
I20260926 05:45:37.452741 12234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "5f25989b8cf6478ca806176020ba426a"
format_stamp: "Formatted at 2026-09-26 05:45:37 on dist-test-slave-1fdt"
server_key: "b2ed58f82f701466d6ff3925efdcaf1b"
server_key_iv: "25e14127acb1f98c5a4742a3fd0387bf"
server_key_version: "encryptionkey@0"
I20260926 05:45:37.454142 12234 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:37.454887 12249 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:37.455047 12234 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:37.455152 12234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "5f25989b8cf6478ca806176020ba426a"
format_stamp: "Formatted at 2026-09-26 05:45:37 on dist-test-slave-1fdt"
server_key: "b2ed58f82f701466d6ff3925efdcaf1b"
server_key_iv: "25e14127acb1f98c5a4742a3fd0387bf"
server_key_version: "encryptionkey@0"
I20260926 05:45:37.455236 12234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:37.466737 12234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:37.468536 12234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:37.468696 12234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:37.472899 12234 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38107
I20260926 05:45:37.472944 12301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38107 every 8 connection(s)
I20260926 05:45:37.473292 12234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:37.473562 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12234
I20260926 05:45:37.473717 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:37.474007 24407 external_mini_cluster.cc:1428] Setting key 98c772d2055a3e4cfcd5130fc5f68531
I20260926 05:45:37.474395 12302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:37.477561 12302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a: Bootstrap starting.
I20260926 05:45:37.478294 12302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:37.478621 12302 log.cc:824] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:37.479387 12302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a: No bootstrap required, opened a new log
I20260926 05:45:37.479566 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:37.474526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50739 (local address 127.0.0.1:38107)
0926 05:45:37.474720 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:37.474729 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:37.474755 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:45:37.475178 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:37.475184 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:37.475235 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:37.475346 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:37.475564 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:37.476422 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:37.477509 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:37.478006 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:37.478091 (+ 85us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:37.478820 (+ 729us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:37.478824 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:37.478824 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:37.478992 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:37.479043 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:37.479093 (+ 50us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:37.479190 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:37.479285 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 05:45:37.479312 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
I20260926 05:45:37.480583 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:37.481956 12302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f25989b8cf6478ca806176020ba426a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260926 05:45:37.482158 12302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:37.482239 12302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f25989b8cf6478ca806176020ba426a, State: Initialized, Role: FOLLOWER
I20260926 05:45:37.482385 12302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f25989b8cf6478ca806176020ba426a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260926 05:45:37.482482 12302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:37.482522 12302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:37.482571 12302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:37.483129 12302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f25989b8cf6478ca806176020ba426a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260926 05:45:37.483254 12302 leader_election.cc:302] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f25989b8cf6478ca806176020ba426a; no voters:
I20260926 05:45:37.483408 12302 leader_election.cc:288] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:37.483455 12312 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:37.483599 12312 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [term 1 LEADER]: Becoming Leader. State: Replica: 5f25989b8cf6478ca806176020ba426a, State: Running, Role: LEADER
I20260926 05:45:37.483659 12302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:37.483762 12312 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f25989b8cf6478ca806176020ba426a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } }
I20260926 05:45:37.484804 12314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f25989b8cf6478ca806176020ba426a. Latest consensus state: current_term: 1 leader_uuid: "5f25989b8cf6478ca806176020ba426a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f25989b8cf6478ca806176020ba426a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } } }
I20260926 05:45:37.484970 12314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:37.485368 12327 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:37.485446 12313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f25989b8cf6478ca806176020ba426a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f25989b8cf6478ca806176020ba426a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38107 } } }
I20260926 05:45:37.485522 12313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:37.485584 12305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:37.482196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34029 (local address 127.0.0.1:38107)
0926 05:45:37.482231 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:37.482233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:37.482335 (+ 102us) server_negotiation.cc:407] Connection header received
0926 05:45:37.482398 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:37.482399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:37.482409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:37.482474 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:37.482611 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:37.483075 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:37.484053 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:37.484334 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:37.484423 (+ 89us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:37.485209 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:37.485210 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:37.485210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:37.485215 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:37.485229 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:37.485233 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:37.485288 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:37.485486 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 05:45:37.485496 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:37.485888 12327 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:37.487351 12327 catalog_manager.cc:1347] Generated new cluster ID: c74c5fcd28304ca0811adc77ceadc450
I20260926 05:45:37.487406 12327 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:37.499158 12327 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:37.499881 12327 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:37.504420 12327 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f25989b8cf6478ca806176020ba426a: Generated new TSK 0
I20260926 05:45:37.504592 12327 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:37.508980 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12234
W20260926 05:45:37.513525 12219 connection.cc:535] client connection to 127.0.0.1:38107 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38107): Connection reset by peer (error 104)
2026-09-26T05:45:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1297 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:37 dist-test-slave-1fdt krb5kdc[12333](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:37 dist-test-slave-1fdt krb5kdc[12333](info): set up 2 sockets
Sep 26 05:45:37 dist-test-slave-1fdt krb5kdc[12333](info): commencing operation
krb5kdc: starting...
W20260926 05:45:38.533597 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.006s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:38 dist-test-slave-1fdt krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401538, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:38.649394 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40087
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:38.729200 12346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:38.729440 12346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:38.729480 12346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:38.730625 12346 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:38.730688 12346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:38.730712 12346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:38.730732 12346 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:38.730762 12346 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:38.732127 12346 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40087
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:38.732398 12346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:38.732609 12346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:38.737370 12352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:38.737452 12346 server_base.cc:1034] running on GCE node
W20260926 05:45:38.737398 12351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:38.737398 12354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:38.737852 12346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:38.738090 12346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:38.739228 12346 hybrid_clock.cc:648] HybridClock initialized: now 1790401538739215 us; error 31 us; skew 500 ppm
Sep 26 05:45:38 dist-test-slave-1fdt krb5kdc[12333](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401538, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:38.741714 12346 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:38.742209 12346 webserver.cc:460] Webserver started at http://127.0.0.1:34843/ using document root <none> and password file <none>
I20260926 05:45:38.742455 12346 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:38.742533 12346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:38.742702 12346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:38.743769 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "8ac92bcc1d694e75b0cf752fb8eb1348"
format_stamp: "Formatted at 2026-09-26 05:45:38 on dist-test-slave-1fdt"
server_key: "89e5a2de821befbec5b33daef748eb96"
server_key_iv: "243a2607d621b3946f09671e88a1c54a"
server_key_version: "encryptionkey@0"
I20260926 05:45:38.744128 12346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "8ac92bcc1d694e75b0cf752fb8eb1348"
format_stamp: "Formatted at 2026-09-26 05:45:38 on dist-test-slave-1fdt"
server_key: "89e5a2de821befbec5b33daef748eb96"
server_key_iv: "243a2607d621b3946f09671e88a1c54a"
server_key_version: "encryptionkey@0"
I20260926 05:45:38.745460 12346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:38.746291 12361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:38.746467 12346 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:38.746565 12346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "8ac92bcc1d694e75b0cf752fb8eb1348"
format_stamp: "Formatted at 2026-09-26 05:45:38 on dist-test-slave-1fdt"
server_key: "89e5a2de821befbec5b33daef748eb96"
server_key_iv: "243a2607d621b3946f09671e88a1c54a"
server_key_version: "encryptionkey@0"
I20260926 05:45:38.746651 12346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:38.749895 12346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:38.751735 12346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:38.751897 12346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:38.755913 12346 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37661
I20260926 05:45:38.755945 12413 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37661 every 8 connection(s)
I20260926 05:45:38.756268 12346 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:38.757270 12414 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:38.759541 12414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348: Bootstrap starting.
I20260926 05:45:38.760126 12414 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:38.760421 12414 log.cc:824] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:38.761108 12414 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348: No bootstrap required, opened a new log
I20260926 05:45:38.762856 12414 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37661 } }
I20260926 05:45:38.762990 12414 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:38.763036 12414 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ac92bcc1d694e75b0cf752fb8eb1348, State: Initialized, Role: FOLLOWER
I20260926 05:45:38.763206 12414 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37661 } }
I20260926 05:45:38.763268 12414 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:38.763345 12414 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:38.763391 12414 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:38.763525 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12346
I20260926 05:45:38.763643 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:38.763819 24407 external_mini_cluster.cc:1428] Setting key a3cf88f4a831c594ef991784dd62c1bc
I20260926 05:45:38.764079 12414 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37661 } }
I20260926 05:45:38.764209 12414 leader_election.cc:302] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ac92bcc1d694e75b0cf752fb8eb1348; no voters:
I20260926 05:45:38.764391 12414 leader_election.cc:288] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:38.764443 12419 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:38.764627 12419 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [term 1 LEADER]: Becoming Leader. State: Replica: 8ac92bcc1d694e75b0cf752fb8eb1348, State: Running, Role: LEADER
I20260926 05:45:38.764726 12414 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:38.764727 12419 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37661 } }
I20260926 05:45:38.765100 12419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37661 } } }
I20260926 05:45:38.765242 12419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:38.765421 12419 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ac92bcc1d694e75b0cf752fb8eb1348. Latest consensus state: current_term: 1 leader_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ac92bcc1d694e75b0cf752fb8eb1348" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37661 } } }
I20260926 05:45:38.765534 12419 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:38.765697 12423 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:38.766350 12423 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:38.766505 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:38.764270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45359 (local address 127.0.0.1:37661)
0926 05:45:38.764472 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:38.764482 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:38.764565 (+ 83us) server_negotiation.cc:407] Connection header received
0926 05:45:38.764725 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:38.764732 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:38.764774 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:38.764914 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:38.765920 (+ 1006us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:38.765923 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:38.765924 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:38.765928 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:38.765969 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:38.765984 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:38.766040 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:38.766278 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 05:45:38.766317 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":90,"threads_started":1}
I20260926 05:45:38.767396 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:38.767989 12423 catalog_manager.cc:1347] Generated new cluster ID: 0a20f8f2d2b94e038922f32e65391f16
I20260926 05:45:38.768028 12423 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:38.770109 12418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:38.769024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39751 (local address 127.0.0.1:37661)
0926 05:45:38.769056 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:38.769058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:38.769146 (+ 88us) server_negotiation.cc:407] Connection header received
0926 05:45:38.769223 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:38.769224 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:38.769228 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:38.769254 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:38.769900 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:38.769901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:38.769901 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:38.769904 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:38.769916 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:38.769919 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:38.769945 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:38.770029 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 05:45:38.770041 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:45:38.795030 12423 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:38.795238 12423 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:38.798518 12423 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ac92bcc1d694e75b0cf752fb8eb1348: Generated new TSK 0
I20260926 05:45:38.798727 12423 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:38.800277 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12346
2026-09-26T05:45:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1291 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:38 dist-test-slave-1fdt krb5kdc[12445](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:38 dist-test-slave-1fdt krb5kdc[12445](info): set up 2 sockets
Sep 26 05:45:38 dist-test-slave-1fdt krb5kdc[12445](info): commencing operation
krb5kdc: starting...
W20260926 05:45:39.837909 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.019s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:39 dist-test-slave-1fdt krb5kdc[12445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:39.956090 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40385
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38581
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40385
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:40.036127 12458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:40.036394 12458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:40.036434 12458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:40.037711 12458 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:40.037791 12458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:40.037815 12458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:40.037842 12458 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:40.037878 12458 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:40.039394 12458 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38581
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:40.039676 12458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:40.039919 12458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:40.044800 12463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:40.044807 12466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:40.044821 12464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:40.045277 12458 server_base.cc:1034] running on GCE node
I20260926 05:45:40.045498 12458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:40.045796 12458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:40.046952 12458 hybrid_clock.cc:648] HybridClock initialized: now 1790401540046926 us; error 56 us; skew 500 ppm
Sep 26 05:45:40 dist-test-slave-1fdt krb5kdc[12445](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401540, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:40.049531 12458 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:40.050048 12458 webserver.cc:460] Webserver started at http://127.0.0.1:45251/ using document root <none> and password file <none>
I20260926 05:45:40.050323 12458 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:40.050387 12458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:40.050523 12458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:40.051580 12458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "cc07795daab045f29121dc41de0a882b"
format_stamp: "Formatted at 2026-09-26 05:45:40 on dist-test-slave-1fdt"
server_key: "0608329acbe73aa9e3d4a1571143746d"
server_key_iv: "528e86e5c7519c0362bfe2bdab6126da"
server_key_version: "encryptionkey@0"
I20260926 05:45:40.051976 12458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "cc07795daab045f29121dc41de0a882b"
format_stamp: "Formatted at 2026-09-26 05:45:40 on dist-test-slave-1fdt"
server_key: "0608329acbe73aa9e3d4a1571143746d"
server_key_iv: "528e86e5c7519c0362bfe2bdab6126da"
server_key_version: "encryptionkey@0"
I20260926 05:45:40.053371 12458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:40.054198 12473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:40.054425 12458 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 05:45:40.054538 12458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "cc07795daab045f29121dc41de0a882b"
format_stamp: "Formatted at 2026-09-26 05:45:40 on dist-test-slave-1fdt"
server_key: "0608329acbe73aa9e3d4a1571143746d"
server_key_iv: "528e86e5c7519c0362bfe2bdab6126da"
server_key_version: "encryptionkey@0"
I20260926 05:45:40.054613 12458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:40.065069 12458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:40.067312 12458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:40.067418 12458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:40.071276 12458 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40385
I20260926 05:45:40.071323 12525 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40385 every 8 connection(s)
I20260926 05:45:40.071674 12458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:40.072517 12526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:40.074819 12526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b: Bootstrap starting.
I20260926 05:45:40.075377 12526 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:40.075752 12526 log.cc:824] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:40.076555 12526 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b: No bootstrap required, opened a new log
I20260926 05:45:40.078352 12526 raft_consensus.cc:348] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc07795daab045f29121dc41de0a882b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40385 } }
I20260926 05:45:40.078544 12526 raft_consensus.cc:374] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:40.078590 12526 raft_consensus.cc:729] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc07795daab045f29121dc41de0a882b, State: Initialized, Role: FOLLOWER
I20260926 05:45:40.078732 12526 consensus_queue.cc:260] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc07795daab045f29121dc41de0a882b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40385 } }
I20260926 05:45:40.078811 12526 raft_consensus.cc:388] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:40.078855 12526 raft_consensus.cc:482] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:40.078904 12526 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:40.079419 12526 raft_consensus.cc:504] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc07795daab045f29121dc41de0a882b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40385 } }
I20260926 05:45:40.079542 12526 leader_election.cc:302] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cc07795daab045f29121dc41de0a882b; no voters:
I20260926 05:45:40.079725 12526 leader_election.cc:288] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:40.079771 12529 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:40.079931 12529 raft_consensus.cc:686] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [term 1 LEADER]: Becoming Leader. State: Replica: cc07795daab045f29121dc41de0a882b, State: Running, Role: LEADER
I20260926 05:45:40.079972 12526 sys_catalog.cc:564] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:40.080070 12529 consensus_queue.cc:237] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc07795daab045f29121dc41de0a882b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40385 } }
I20260926 05:45:40.080475 12530 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cc07795daab045f29121dc41de0a882b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc07795daab045f29121dc41de0a882b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40385 } } }
I20260926 05:45:40.080466 12531 sys_catalog.cc:455] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [sys.catalog]: SysCatalogTable state changed. Reason: New leader cc07795daab045f29121dc41de0a882b. Latest consensus state: current_term: 1 leader_uuid: "cc07795daab045f29121dc41de0a882b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cc07795daab045f29121dc41de0a882b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40385 } } }
I20260926 05:45:40.080560 12530 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:40.080566 12531 sys_catalog.cc:458] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:40.080752 12533 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:40.080907 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12458
I20260926 05:45:40.081060 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:40.081264 24407 external_mini_cluster.cc:1428] Setting key 2c2218b0e1cd1083c9fe8b7d3b695e47
I20260926 05:45:40.081338 12533 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:40.086117 12533 catalog_manager.cc:1347] Generated new cluster ID: af5266b413bc4d4a8f671a8b83079f23
I20260926 05:45:40.086189 12533 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:45:40 dist-test-slave-1fdt krb5kdc[12445](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401539, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:45:40.091233 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:40.081945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47303 (local address 127.0.0.1:40385)
0926 05:45:40.082106 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:40.082115 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:40.082297 (+ 182us) server_negotiation.cc:407] Connection header received
0926 05:45:40.082448 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:40.082455 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:40.082497 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:40.082624 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:40.083255 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:40.084147 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:40.084912 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:40.085348 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:40.085466 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:40.087931 (+ 2465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:40.087935 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:40.087935 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:40.087946 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:40.089645 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:40.090083 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:40.090086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:40.090086 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:40.090187 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:40.090406 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:40.090407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:40.090407 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:40.090639 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:40.090819 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:40.090896 (+ 77us) server_negotiation.cc:299] Negotiation successful
0926 05:45:40.091049 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":65,"threads_started":1}
I20260926 05:45:40.092151 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:40.092932 12533 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:40.093634 12533 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:40.099779 12547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:40.093672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42797 (local address 127.0.0.1:40385)
0926 05:45:40.093710 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:40.093712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:40.093838 (+ 126us) server_negotiation.cc:407] Connection header received
0926 05:45:40.093908 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:40.093909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:40.093919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:40.093973 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:40.094457 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:40.094967 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:40.095700 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:40.095914 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:40.095985 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:40.097124 (+ 1139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:40.097125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:40.097125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:40.097131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:40.098628 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:40.098982 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:40.098983 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:40.098984 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:40.099035 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:40.099250 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:40.099251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:40.099252 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:40.099358 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:40.099465 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:40.099590 (+ 125us) server_negotiation.cc:299] Negotiation successful
0926 05:45:40.099663 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:45:40.105145 12533 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cc07795daab045f29121dc41de0a882b: Generated new TSK 0
I20260926 05:45:40.105365 12533 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:40.109351 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12458
2026-09-26T05:45:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1308 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:40 dist-test-slave-1fdt krb5kdc[12557](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:40 dist-test-slave-1fdt krb5kdc[12557](info): set up 2 sockets
Sep 26 05:45:40 dist-test-slave-1fdt krb5kdc[12557](info): commencing operation
krb5kdc: starting...
W20260926 05:45:41.150206 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:41 dist-test-slave-1fdt krb5kdc[12557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401541, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:41.268976 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42835
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:41.349086 12570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:41.349335 12570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:41.349372 12570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:41.350709 12570 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:41.350793 12570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:41.350816 12570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:41.350837 12570 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:41.350867 12570 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:41.352378 12570 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42835
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:41.352672 12570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:41.352908 12570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 05:45:41.357726 12570 server_base.cc:1034] running on GCE node
W20260926 05:45:41.357712 12576 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:41.357726 12575 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:41.357854 12578 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:41.358157 12570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:41.358457 12570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:41.359622 12570 hybrid_clock.cc:648] HybridClock initialized: now 1790401541359602 us; error 53 us; skew 500 ppm
Sep 26 05:45:41 dist-test-slave-1fdt krb5kdc[12557](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401541, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:41.365262 12570 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:41.365705 12570 webserver.cc:460] Webserver started at http://127.0.0.1:35147/ using document root <none> and password file <none>
I20260926 05:45:41.365948 12570 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:41.365993 12570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:41.366080 12570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:41.367064 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "5bc48b5955c04672ac91ad8dca543043"
format_stamp: "Formatted at 2026-09-26 05:45:41 on dist-test-slave-1fdt"
server_key: "c8c471d3122c2fea8849e028a79b7949"
server_key_iv: "3dd9ce1c6337fd38b623274c06229fd2"
server_key_version: "encryptionkey@0"
I20260926 05:45:41.367380 12570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "5bc48b5955c04672ac91ad8dca543043"
format_stamp: "Formatted at 2026-09-26 05:45:41 on dist-test-slave-1fdt"
server_key: "c8c471d3122c2fea8849e028a79b7949"
server_key_iv: "3dd9ce1c6337fd38b623274c06229fd2"
server_key_version: "encryptionkey@0"
I20260926 05:45:41.368628 12570 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:41.369302 12585 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:41.369515 12570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:41.369602 12570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "5bc48b5955c04672ac91ad8dca543043"
format_stamp: "Formatted at 2026-09-26 05:45:41 on dist-test-slave-1fdt"
server_key: "c8c471d3122c2fea8849e028a79b7949"
server_key_iv: "3dd9ce1c6337fd38b623274c06229fd2"
server_key_version: "encryptionkey@0"
I20260926 05:45:41.369669 12570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:41.401024 12570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:41.402972 12570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:41.403087 12570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:41.406989 12570 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37097
I20260926 05:45:41.407044 12637 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:37097 every 8 connection(s)
I20260926 05:45:41.407390 12570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:41.408283 12638 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:41.410615 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043: Bootstrap starting.
I20260926 05:45:41.411201 12638 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:41.411492 12638 log.cc:824] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:41.412173 12638 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043: No bootstrap required, opened a new log
I20260926 05:45:41.413770 12638 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc48b5955c04672ac91ad8dca543043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260926 05:45:41.413932 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12570
I20260926 05:45:41.413965 12638 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:41.414009 12638 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bc48b5955c04672ac91ad8dca543043, State: Initialized, Role: FOLLOWER
I20260926 05:45:41.414043 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:41.414222 24407 external_mini_cluster.cc:1428] Setting key e2ee5bf9380605c0a263ca028db15363
I20260926 05:45:41.414192 12638 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc48b5955c04672ac91ad8dca543043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260926 05:45:41.414268 12638 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:41.414314 12638 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:41.414366 12638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:41.415303 12638 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc48b5955c04672ac91ad8dca543043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260926 05:45:41.415426 12638 leader_election.cc:302] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bc48b5955c04672ac91ad8dca543043; no voters:
I20260926 05:45:41.415589 12638 leader_election.cc:288] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:41.415621 12643 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:41.415817 12643 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [term 1 LEADER]: Becoming Leader. State: Replica: 5bc48b5955c04672ac91ad8dca543043, State: Running, Role: LEADER
I20260926 05:45:41.415833 12638 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:41.415975 12643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc48b5955c04672ac91ad8dca543043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } }
I20260926 05:45:41.417040 12644 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bc48b5955c04672ac91ad8dca543043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc48b5955c04672ac91ad8dca543043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260926 05:45:41.417176 12644 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:41.417367 12658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:41.417387 12645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bc48b5955c04672ac91ad8dca543043. Latest consensus state: current_term: 1 leader_uuid: "5bc48b5955c04672ac91ad8dca543043" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bc48b5955c04672ac91ad8dca543043" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37097 } } }
I20260926 05:45:41.417711 12645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:41.418151 12658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:41.419697 12658 catalog_manager.cc:1347] Generated new cluster ID: ff909d9a42784ea59e0b132afe918bf2
I20260926 05:45:41.419749 12658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:41.420049 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:41.414709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33243 (local address 127.0.0.1:37097)
0926 05:45:41.414947 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:41.414956 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:41.414994 (+ 38us) server_negotiation.cc:407] Connection header received
0926 05:45:41.415096 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:41.415101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:41.415145 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:41.415300 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:41.415504 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:41.416900 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:41.418036 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:41.418524 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:41.418642 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:41.419372 (+ 730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:41.419376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:41.419376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:41.419565 (+ 189us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:41.419629 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:41.419646 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:41.419705 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:41.419821 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 05:45:41.419885 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":123,"threads_started":1}
I20260926 05:45:41.422824 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:41.428706 12642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:41.424196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58227 (local address 127.0.0.1:37097)
0926 05:45:41.424234 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:41.424236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:41.425906 (+ 1670us) server_negotiation.cc:407] Connection header received
0926 05:45:41.425979 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:41.425980 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:41.425991 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:41.426062 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:41.426183 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:41.426648 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:41.427452 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:41.427660 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:41.427753 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:41.428439 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:41.428441 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:41.428441 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:41.428446 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:41.428461 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:41.428465 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:41.428518 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:41.428580 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 05:45:41.428604 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:41.428933 12658 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:41.429622 12658 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:41.433801 12658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bc48b5955c04672ac91ad8dca543043: Generated new TSK 0
I20260926 05:45:41.433996 12658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:41.438699 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12570
2026-09-26T05:45:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:41 dist-test-slave-1fdt krb5kdc[12669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:41 dist-test-slave-1fdt krb5kdc[12669](info): set up 2 sockets
Sep 26 05:45:41 dist-test-slave-1fdt krb5kdc[12669](info): commencing operation
krb5kdc: starting...
W20260926 05:45:42.465960 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.009s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:42 dist-test-slave-1fdt krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401542, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:42.584388 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40485
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34775
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40485
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:42.664060 12682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:42.664261 12682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:42.664284 12682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:42.665486 12682 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:42.665527 12682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:42.665541 12682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:42.665586 12682 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:42.665613 12682 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:42.667181 12682 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34775
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:42.667474 12682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:42.667734 12682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:42.672500 12687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:42.672513 12690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:42.672513 12688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:42.672829 12682 server_base.cc:1034] running on GCE node
I20260926 05:45:42.673077 12682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:42.673404 12682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:42.674569 12682 hybrid_clock.cc:648] HybridClock initialized: now 1790401542674554 us; error 41 us; skew 500 ppm
Sep 26 05:45:42 dist-test-slave-1fdt krb5kdc[12669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401542, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:42.677090 12682 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:42.677521 12682 webserver.cc:460] Webserver started at http://127.0.0.1:37371/ using document root <none> and password file <none>
I20260926 05:45:42.677819 12682 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:42.677889 12682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:42.678048 12682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:42.679184 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "a1cbbd22319c4d94a124b335a5305dd2"
format_stamp: "Formatted at 2026-09-26 05:45:42 on dist-test-slave-1fdt"
server_key: "805d42116269bb086d2a703eddc88986"
server_key_iv: "9ec1d09455a4cc85056aee76879a1d1a"
server_key_version: "encryptionkey@0"
I20260926 05:45:42.679551 12682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "a1cbbd22319c4d94a124b335a5305dd2"
format_stamp: "Formatted at 2026-09-26 05:45:42 on dist-test-slave-1fdt"
server_key: "805d42116269bb086d2a703eddc88986"
server_key_iv: "9ec1d09455a4cc85056aee76879a1d1a"
server_key_version: "encryptionkey@0"
I20260926 05:45:42.680861 12682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:42.681528 12697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:42.681684 12682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:45:42.681798 12682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "a1cbbd22319c4d94a124b335a5305dd2"
format_stamp: "Formatted at 2026-09-26 05:45:42 on dist-test-slave-1fdt"
server_key: "805d42116269bb086d2a703eddc88986"
server_key_iv: "9ec1d09455a4cc85056aee76879a1d1a"
server_key_version: "encryptionkey@0"
I20260926 05:45:42.681880 12682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:42.685251 12682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:42.687132 12682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:42.687287 12682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:42.691183 12682 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40485
I20260926 05:45:42.691231 12749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40485 every 8 connection(s)
I20260926 05:45:42.691572 12682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:42.692606 12750 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:42.695370 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2: Bootstrap starting.
I20260926 05:45:42.695894 12750 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:42.696130 12750 log.cc:824] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:42.696782 12750 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2: No bootstrap required, opened a new log
I20260926 05:45:42.698400 12750 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbbd22319c4d94a124b335a5305dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260926 05:45:42.698534 12750 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:42.698555 12750 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1cbbd22319c4d94a124b335a5305dd2, State: Initialized, Role: FOLLOWER
I20260926 05:45:42.698655 12750 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbbd22319c4d94a124b335a5305dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260926 05:45:42.698704 12750 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:42.698728 12750 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:42.698755 12750 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:42.698925 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12682
I20260926 05:45:42.699049 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:42.699242 24407 external_mini_cluster.cc:1428] Setting key aa77683b4843912247005a14f7e2a3ac
I20260926 05:45:42.699446 12750 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbbd22319c4d94a124b335a5305dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260926 05:45:42.699561 12750 leader_election.cc:302] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1cbbd22319c4d94a124b335a5305dd2; no voters:
I20260926 05:45:42.699733 12750 leader_election.cc:288] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:42.699781 12754 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:42.700031 12754 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [term 1 LEADER]: Becoming Leader. State: Replica: a1cbbd22319c4d94a124b335a5305dd2, State: Running, Role: LEADER
I20260926 05:45:42.700089 12750 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:42.700155 12754 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbbd22319c4d94a124b335a5305dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } }
I20260926 05:45:42.700484 12754 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1cbbd22319c4d94a124b335a5305dd2. Latest consensus state: current_term: 1 leader_uuid: "a1cbbd22319c4d94a124b335a5305dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbbd22319c4d94a124b335a5305dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } } }
I20260926 05:45:42.700616 12754 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:42.700686 12757 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1cbbd22319c4d94a124b335a5305dd2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1cbbd22319c4d94a124b335a5305dd2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40485 } } }
I20260926 05:45:42.700858 12757 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:42.701421 12759 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:42.701825 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:42.699888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51495 (local address 127.0.0.1:40485)
0926 05:45:42.700041 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:42.700051 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:42.700085 (+ 34us) server_negotiation.cc:407] Connection header received
0926 05:45:42.700213 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:42.700221 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:42.700262 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:42.700404 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:42.701125 (+ 721us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:42.701129 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:42.701129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:42.701133 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:42.701181 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:42.701199 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:42.701276 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:42.701550 (+ 274us) server_negotiation.cc:299] Negotiation successful
0926 05:45:42.701593 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":69,"threads_started":1}
I20260926 05:45:42.701952 12759 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:42.702898 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:42.703581 12759 catalog_manager.cc:1347] Generated new cluster ID: 60a32192a62f4b47a7652bbe0d4770ef
I20260926 05:45:42.703624 12759 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:42.710840 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:42.709805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57777 (local address 127.0.0.1:40485)
0926 05:45:42.709844 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:42.709846 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:42.709855 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:45:42.709877 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:42.709878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:42.709883 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:42.709950 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:42.710594 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:42.710595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:42.710596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:42.710600 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:42.710615 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:42.710619 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:42.710668 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:42.710732 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 05:45:42.710746 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 05:45:42.714905 12759 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:42.715080 12759 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:42.723659 12759 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1cbbd22319c4d94a124b335a5305dd2: Generated new TSK 0
I20260926 05:45:42.723862 12759 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:42.732009 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12682
2026-09-26T05:45:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1293 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:42 dist-test-slave-1fdt krb5kdc[12781](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:42 dist-test-slave-1fdt krb5kdc[12781](info): set up 2 sockets
Sep 26 05:45:42 dist-test-slave-1fdt krb5kdc[12781](info): commencing operation
krb5kdc: starting...
W20260926 05:45:43.769830 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:43 dist-test-slave-1fdt krb5kdc[12781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:43.888405 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41321
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41323
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41321
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:43.970269 12794 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:43.970476 12794 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:43.970499 12794 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:43.971611 12794 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:43.971645 12794 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:43.971659 12794 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:43.971671 12794 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:43.971683 12794 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:43.973155 12794 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41323
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41321
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41321
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12794
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:43.973366 12794 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:43.973537 12794 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:43.978727 12799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:43.978775 12802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:43.978731 12800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:43.978747 12794 server_base.cc:1034] running on GCE node
I20260926 05:45:43.979208 12794 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:43.979522 12794 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:43.980667 12794 hybrid_clock.cc:648] HybridClock initialized: now 1790401543980646 us; error 38 us; skew 500 ppm
Sep 26 05:45:43 dist-test-slave-1fdt krb5kdc[12781](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401543, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:43.983397 12794 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:43.983844 12794 webserver.cc:460] Webserver started at http://127.0.0.1:39299/ using document root <none> and password file <none>
I20260926 05:45:43.984092 12794 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:43.984140 12794 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:43.984272 12794 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:43.985363 12794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "1b2a9db9e2134333a145f5b6124be9b3"
format_stamp: "Formatted at 2026-09-26 05:45:43 on dist-test-slave-1fdt"
server_key: "8778b959293fcc2222edfaf4012b018a"
server_key_iv: "d9bfd4bd53d90694f8837be3932094be"
server_key_version: "encryptionkey@0"
I20260926 05:45:43.985724 12794 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "1b2a9db9e2134333a145f5b6124be9b3"
format_stamp: "Formatted at 2026-09-26 05:45:43 on dist-test-slave-1fdt"
server_key: "8778b959293fcc2222edfaf4012b018a"
server_key_iv: "d9bfd4bd53d90694f8837be3932094be"
server_key_version: "encryptionkey@0"
I20260926 05:45:43.987192 12794 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:43.987968 12809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:43.988265 12794 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:43.988339 12794 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "1b2a9db9e2134333a145f5b6124be9b3"
format_stamp: "Formatted at 2026-09-26 05:45:43 on dist-test-slave-1fdt"
server_key: "8778b959293fcc2222edfaf4012b018a"
server_key_iv: "d9bfd4bd53d90694f8837be3932094be"
server_key_version: "encryptionkey@0"
I20260926 05:45:43.988431 12794 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:44.022514 12794 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:44.024964 12794 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:44.025079 12794 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:44.029320 12794 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41321
I20260926 05:45:44.029403 12861 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41321 every 8 connection(s)
I20260926 05:45:44.029827 12794 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:44.030848 12862 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:44.033201 12862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3: Bootstrap starting.
I20260926 05:45:44.033924 12862 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:44.033926 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12794
I20260926 05:45:44.034035 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:44.034248 24407 external_mini_cluster.cc:1428] Setting key ad5293730315e60808c7d0de2b012ba0
I20260926 05:45:44.034314 12862 log.cc:824] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:44.035176 12862 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3: No bootstrap required, opened a new log
I20260926 05:45:44.037186 12862 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2a9db9e2134333a145f5b6124be9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260926 05:45:44.037350 12862 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:44.037400 12862 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b2a9db9e2134333a145f5b6124be9b3, State: Initialized, Role: FOLLOWER
I20260926 05:45:44.037532 12862 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2a9db9e2134333a145f5b6124be9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260926 05:45:44.037616 12862 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:44.037659 12862 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:44.037737 12862 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:44.038355 12862 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2a9db9e2134333a145f5b6124be9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260926 05:45:44.038478 12862 leader_election.cc:302] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b2a9db9e2134333a145f5b6124be9b3; no voters:
I20260926 05:45:44.038693 12862 leader_election.cc:288] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:44.038708 12867 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:44.038862 12867 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [term 1 LEADER]: Becoming Leader. State: Replica: 1b2a9db9e2134333a145f5b6124be9b3, State: Running, Role: LEADER
I20260926 05:45:44.039011 12867 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2a9db9e2134333a145f5b6124be9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } }
I20260926 05:45:44.039009 12862 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:44.039435 12868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b2a9db9e2134333a145f5b6124be9b3. Latest consensus state: current_term: 1 leader_uuid: "1b2a9db9e2134333a145f5b6124be9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2a9db9e2134333a145f5b6124be9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } } }
I20260926 05:45:44.039531 12868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:44.039630 12869 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b2a9db9e2134333a145f5b6124be9b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b2a9db9e2134333a145f5b6124be9b3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41321 } } }
I20260926 05:45:44.039778 12869 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:44.039975 12872 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:44.040526 12872 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 05:45:44 dist-test-slave-1fdt krb5kdc[12781](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401543, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:45:44.042367 12872 catalog_manager.cc:1347] Generated new cluster ID: 262ba48a4f2a43859276b44fbd4f4df6
I20260926 05:45:44.042434 12872 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:44.045501 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:44.034801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:33177 (local address 127.0.0.1:41321)
0926 05:45:44.034959 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:44.034970 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:44.035006 (+ 36us) server_negotiation.cc:407] Connection header received
0926 05:45:44.035532 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:44.035538 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:44.035590 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:44.035744 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:44.036357 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:44.037248 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:44.038203 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:44.038713 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:44.038845 (+ 132us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:44.041682 (+ 2837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:44.041686 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:44.041687 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:44.041699 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:44.043752 (+ 2053us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:44.044223 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:44.044226 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:44.044226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:44.044338 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:44.044573 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:44.044574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:44.044575 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:44.044893 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:44.045067 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:44.045160 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 05:45:44.045276 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":71,"threads_started":1}
I20260926 05:45:44.046558 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:44.046902 12872 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:44.047566 12872 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:44.055431 12866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:44.048292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:36739 (local address 127.0.0.1:41321)
0926 05:45:44.048328 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:44.048330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:44.048338 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:45:44.048361 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:44.048362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:44.048371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:44.048436 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:44.048992 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:44.049820 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:44.050847 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:44.051184 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:44.051256 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:44.052590 (+ 1334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:44.052592 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:44.052592 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:44.052599 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:44.054217 (+ 1618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:44.054617 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:44.054618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:44.054619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:44.054680 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:44.054906 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:44.054906 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:44.054907 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:44.055036 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:44.055156 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:44.055272 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 05:45:44.055337 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:45:44.059283 12872 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1b2a9db9e2134333a145f5b6124be9b3: Generated new TSK 0
I20260926 05:45:44.059476 12872 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:44.063278 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12794
W20260926 05:45:44.067659 12779 connection.cc:535] client connection to 127.0.0.1:41321 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41321): Connection reset by peer (error 104)
2026-09-26T05:45:44Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1331 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:44 dist-test-slave-1fdt krb5kdc[12893](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:44 dist-test-slave-1fdt krb5kdc[12893](info): set up 2 sockets
Sep 26 05:45:44 dist-test-slave-1fdt krb5kdc[12893](info): commencing operation
krb5kdc: starting...
W20260926 05:45:45.110074 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.029s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:45 dist-test-slave-1fdt krb5kdc[12893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401545, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:45.228231 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40757
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44531
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40757
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:45.308516 12906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:45.308714 12906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:45.308735 12906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:45.309849 12906 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:45.309890 12906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:45.309903 12906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:45.309916 12906 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:45.309928 12906 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:45.311301 12906 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44531
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40757
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12906
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:45.311479 12906 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:45.311653 12906 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:45.316476 12912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:45.316463 12911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:45.316486 12914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:45.316637 12906 server_base.cc:1034] running on GCE node
I20260926 05:45:45.316871 12906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:45.317157 12906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:45.318302 12906 hybrid_clock.cc:648] HybridClock initialized: now 1790401545318269 us; error 45 us; skew 500 ppm
Sep 26 05:45:45 dist-test-slave-1fdt krb5kdc[12893](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401545, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:45.320719 12906 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:45.321086 12906 webserver.cc:460] Webserver started at http://127.0.0.1:45491/ using document root <none> and password file <none>
I20260926 05:45:45.321270 12906 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:45.321310 12906 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:45.321395 12906 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:45.322439 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "3294f19fe6854e9daaa6522d88652925"
format_stamp: "Formatted at 2026-09-26 05:45:45 on dist-test-slave-1fdt"
server_key: "8f9ee6ec118b032c5d4ec42f0c0cc16d"
server_key_iv: "d91c2df07209b7c006d0e179519b5bb1"
server_key_version: "encryptionkey@0"
I20260926 05:45:45.322826 12906 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "3294f19fe6854e9daaa6522d88652925"
format_stamp: "Formatted at 2026-09-26 05:45:45 on dist-test-slave-1fdt"
server_key: "8f9ee6ec118b032c5d4ec42f0c0cc16d"
server_key_iv: "d91c2df07209b7c006d0e179519b5bb1"
server_key_version: "encryptionkey@0"
I20260926 05:45:45.324060 12906 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:45.324867 12921 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:45.325088 12906 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:45.325206 12906 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "3294f19fe6854e9daaa6522d88652925"
format_stamp: "Formatted at 2026-09-26 05:45:45 on dist-test-slave-1fdt"
server_key: "8f9ee6ec118b032c5d4ec42f0c0cc16d"
server_key_iv: "d91c2df07209b7c006d0e179519b5bb1"
server_key_version: "encryptionkey@0"
I20260926 05:45:45.325282 12906 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:45.337849 12906 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:45.340180 12906 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:45.340355 12906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:45.344566 12906 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40757
I20260926 05:45:45.344630 12973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40757 every 8 connection(s)
I20260926 05:45:45.345010 12906 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:45.345940 12974 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:45.348227 12974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925: Bootstrap starting.
I20260926 05:45:45.348767 12974 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:45.349027 12974 log.cc:824] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:45.349706 12974 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925: No bootstrap required, opened a new log
I20260926 05:45:45.351504 12974 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294f19fe6854e9daaa6522d88652925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260926 05:45:45.351657 12974 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:45.351680 12974 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3294f19fe6854e9daaa6522d88652925, State: Initialized, Role: FOLLOWER
I20260926 05:45:45.351823 12974 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294f19fe6854e9daaa6522d88652925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260926 05:45:45.351905 12974 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:45.351928 12974 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:45.351958 12974 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:45.352465 12974 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294f19fe6854e9daaa6522d88652925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260926 05:45:45.352556 12974 leader_election.cc:302] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3294f19fe6854e9daaa6522d88652925; no voters:
I20260926 05:45:45.352681 12974 leader_election.cc:288] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:45.352741 12977 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:45.352916 12977 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [term 1 LEADER]: Becoming Leader. State: Replica: 3294f19fe6854e9daaa6522d88652925, State: Running, Role: LEADER
I20260926 05:45:45.352958 12974 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:45.353015 12977 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294f19fe6854e9daaa6522d88652925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } }
I20260926 05:45:45.353274 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 12906
I20260926 05:45:45.353381 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:45.353596 24407 external_mini_cluster.cc:1428] Setting key a5b4ccc63ba129067764ee052626eb47
I20260926 05:45:45.354208 12978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3294f19fe6854e9daaa6522d88652925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294f19fe6854e9daaa6522d88652925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } } }
I20260926 05:45:45.354300 12978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:45.354384 12979 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3294f19fe6854e9daaa6522d88652925. Latest consensus state: current_term: 1 leader_uuid: "3294f19fe6854e9daaa6522d88652925" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3294f19fe6854e9daaa6522d88652925" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40757 } } }
I20260926 05:45:45.354528 12979 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925 [sys.catalog]: This master's current role is: LEADER
W20260926 05:45:45.355132 12995 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:45:45.355194 12995 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:45:45.355263 12994 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:45.355456 12994 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:45.357066 12994 catalog_manager.cc:1347] Generated new cluster ID: 00c6ee3b9b2b4102a47fa95754f37fe2
I20260926 05:45:45.357131 12994 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:45.359809 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:45.354288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:52847 (local address 127.0.0.1:40757)
0926 05:45:45.354503 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:45.354600 (+ 97us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:45.354716 (+ 116us) server_negotiation.cc:407] Connection header received
0926 05:45:45.354857 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:45.354863 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:45.354931 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:45.355086 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:45.355334 (+ 248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:45.356593 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:45.357688 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:45.358301 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:45.358401 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:45.359144 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:45.359148 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:45.359149 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:45.359359 (+ 210us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:45.359394 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:45.359407 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:45.359484 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:45.359589 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 05:45:45.359613 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":78,"threads_started":1}
I20260926 05:45:45.360723 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:45.366888 12993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:45.363485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:60809 (local address 127.0.0.1:40757)
0926 05:45:45.363520 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:45.363521 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:45.363581 (+ 60us) server_negotiation.cc:407] Connection header received
0926 05:45:45.363639 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:45.363640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:45.363650 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:45.363726 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:45.363818 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:45.364541 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:45.365519 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:45.365846 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:45.365912 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:45.366635 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:45.366636 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:45.366636 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:45.366640 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:45.366655 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:45.366659 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:45.366730 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:45.366795 (+ 65us) server_negotiation.cc:299] Negotiation successful
0926 05:45:45.366808 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:45:45.369827 12994 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:45.370419 12994 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:45.379128 12994 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3294f19fe6854e9daaa6522d88652925: Generated new TSK 0
I20260926 05:45:45.379318 12994 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:45.385882 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 12906
W20260926 05:45:45.390302 12891 connection.cc:535] client connection to 127.0.0.1:40757 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40757): Connection reset by peer (error 104)
2026-09-26T05:45:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1322 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:45 dist-test-slave-1fdt krb5kdc[13005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:45 dist-test-slave-1fdt krb5kdc[13005](info): set up 2 sockets
Sep 26 05:45:45 dist-test-slave-1fdt krb5kdc[13005](info): commencing operation
krb5kdc: starting...
W20260926 05:45:46.425971 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.022s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:46 dist-test-slave-1fdt krb5kdc[13005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401546, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:46.542608 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45707
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38977
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45707
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:46.623986 13018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:46.624243 13018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:46.624284 13018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:46.625432 13018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:46.625504 13018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:46.625535 13018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:46.625578 13018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:46.625608 13018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:46.627121 13018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38977
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45707
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45707
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13018
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:46.627410 13018 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:46.627642 13018 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:46.632445 13024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:46.632429 13026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:46.632483 13023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:46.632671 13018 server_base.cc:1034] running on GCE node
I20260926 05:45:46.632941 13018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:46.633215 13018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:46.634371 13018 hybrid_clock.cc:648] HybridClock initialized: now 1790401546634359 us; error 45 us; skew 500 ppm
Sep 26 05:45:46 dist-test-slave-1fdt krb5kdc[13005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401546, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:46.643347 13018 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:46.643851 13018 webserver.cc:460] Webserver started at http://127.0.0.1:43451/ using document root <none> and password file <none>
I20260926 05:45:46.644080 13018 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:46.644150 13018 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:46.644299 13018 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:46.645383 13018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "95bc60b111d948b58321a6f42259e32d"
format_stamp: "Formatted at 2026-09-26 05:45:46 on dist-test-slave-1fdt"
server_key: "ffc981024673bf2b401863aa634587a2"
server_key_iv: "78d84227194f150ad1aaebe1d2ad34e3"
server_key_version: "encryptionkey@0"
I20260926 05:45:46.645859 13018 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "95bc60b111d948b58321a6f42259e32d"
format_stamp: "Formatted at 2026-09-26 05:45:46 on dist-test-slave-1fdt"
server_key: "ffc981024673bf2b401863aa634587a2"
server_key_iv: "78d84227194f150ad1aaebe1d2ad34e3"
server_key_version: "encryptionkey@0"
I20260926 05:45:46.647212 13018 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:46.647970 13033 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:46.648145 13018 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:46.648250 13018 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "95bc60b111d948b58321a6f42259e32d"
format_stamp: "Formatted at 2026-09-26 05:45:46 on dist-test-slave-1fdt"
server_key: "ffc981024673bf2b401863aa634587a2"
server_key_iv: "78d84227194f150ad1aaebe1d2ad34e3"
server_key_version: "encryptionkey@0"
I20260926 05:45:46.648336 13018 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:46.651705 13018 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:46.654062 13018 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:46.654217 13018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:46.658241 13018 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45707
I20260926 05:45:46.658270 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45707 every 8 connection(s)
I20260926 05:45:46.658591 13018 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:46.659571 13086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:46.661834 13086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d: Bootstrap starting.
I20260926 05:45:46.662408 13086 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:46.662686 13086 log.cc:824] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:46.663404 13086 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d: No bootstrap required, opened a new log
I20260926 05:45:46.664963 13086 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bc60b111d948b58321a6f42259e32d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260926 05:45:46.665097 13086 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:46.665162 13086 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95bc60b111d948b58321a6f42259e32d, State: Initialized, Role: FOLLOWER
I20260926 05:45:46.665318 13086 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bc60b111d948b58321a6f42259e32d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260926 05:45:46.665377 13086 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:46.665426 13086 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:46.665499 13086 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:46.666139 13086 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bc60b111d948b58321a6f42259e32d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260926 05:45:46.666227 13086 leader_election.cc:302] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95bc60b111d948b58321a6f42259e32d; no voters:
I20260926 05:45:46.666432 13086 leader_election.cc:288] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:46.666486 13089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:46.666671 13089 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [term 1 LEADER]: Becoming Leader. State: Replica: 95bc60b111d948b58321a6f42259e32d, State: Running, Role: LEADER
I20260926 05:45:46.666688 13086 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:46.666761 13089 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bc60b111d948b58321a6f42259e32d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } }
I20260926 05:45:46.667131 13090 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95bc60b111d948b58321a6f42259e32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bc60b111d948b58321a6f42259e32d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } } }
I20260926 05:45:46.667140 13091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95bc60b111d948b58321a6f42259e32d. Latest consensus state: current_term: 1 leader_uuid: "95bc60b111d948b58321a6f42259e32d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95bc60b111d948b58321a6f42259e32d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45707 } } }
I20260926 05:45:46.667249 13090 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:46.667265 13091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:46.667454 13093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:46.667647 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13018
I20260926 05:45:46.667788 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:46.667975 13093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:46.667989 24407 external_mini_cluster.cc:1428] Setting key d5e3ab286c5995016a324980496fad88
I20260926 05:45:46.669816 13093 catalog_manager.cc:1347] Generated new cluster ID: e16b8e0800994d9391cd16a611fb5742
I20260926 05:45:46.669867 13093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:46.674803 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:46.668593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:55191 (local address 127.0.0.1:45707)
0926 05:45:46.668876 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:46.668885 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:46.668917 (+ 32us) server_negotiation.cc:407] Connection header received
0926 05:45:46.669031 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:46.669038 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:46.669068 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:46.669222 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:46.674311 (+ 5089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:46.674316 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:46.674316 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:46.674321 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:46.674372 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:46.674385 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:46.674463 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:46.674538 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 05:45:46.674590 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":234,"thread_start_us":83,"threads_started":1}
I20260926 05:45:46.675758 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:46.678988 13108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:46.677188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:46605 (local address 127.0.0.1:45707)
0926 05:45:46.677225 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:46.677227 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:46.677950 (+ 723us) server_negotiation.cc:407] Connection header received
0926 05:45:46.678048 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:46.678049 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:46.678065 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:46.678127 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:46.678758 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:46.678760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:46.678760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:46.678765 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:46.678780 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:46.678785 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:46.678830 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:46.678882 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 05:45:46.678898 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:45:46.680671 13093 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:46.680831 13093 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:46.683873 13093 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95bc60b111d948b58321a6f42259e32d: Generated new TSK 0
I20260926 05:45:46.684060 13093 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:46.685663 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13018
2026-09-26T05:45:46Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1299 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:46 dist-test-slave-1fdt krb5kdc[13117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:46 dist-test-slave-1fdt krb5kdc[13117](info): set up 2 sockets
Sep 26 05:45:46 dist-test-slave-1fdt krb5kdc[13117](info): commencing operation
krb5kdc: starting...
W20260926 05:45:47.718086 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:47 dist-test-slave-1fdt krb5kdc[13117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:47.834584 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35849
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35849
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:47.914354 13130 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:47.914618 13130 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:47.914659 13130 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:47.915799 13130 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:47.915874 13130 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:47.915897 13130 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:47.915927 13130 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:47.915957 13130 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:47.917397 13130 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35849
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13130
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:47.917677 13130 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:47.917938 13130 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:47.922638 13135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:47.922610 13138 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:47.922664 13136 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:47.922950 13130 server_base.cc:1034] running on GCE node
I20260926 05:45:47.923233 13130 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:47.923518 13130 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:47.924700 13130 hybrid_clock.cc:648] HybridClock initialized: now 1790401547924692 us; error 41 us; skew 500 ppm
Sep 26 05:45:47 dist-test-slave-1fdt krb5kdc[13117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401547, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:47.927284 13130 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:47.927723 13130 webserver.cc:460] Webserver started at http://127.0.0.1:33401/ using document root <none> and password file <none>
I20260926 05:45:47.927956 13130 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:47.928031 13130 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:47.928177 13130 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:47.929252 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "9a6105776d434995b5bd2796df70c534"
format_stamp: "Formatted at 2026-09-26 05:45:47 on dist-test-slave-1fdt"
server_key: "c0e45ccf40f8dbff48e5194fabcd897e"
server_key_iv: "8fd14344f21ef93d9a7bb7b1a1476c28"
server_key_version: "encryptionkey@0"
I20260926 05:45:47.929610 13130 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "9a6105776d434995b5bd2796df70c534"
format_stamp: "Formatted at 2026-09-26 05:45:47 on dist-test-slave-1fdt"
server_key: "c0e45ccf40f8dbff48e5194fabcd897e"
server_key_iv: "8fd14344f21ef93d9a7bb7b1a1476c28"
server_key_version: "encryptionkey@0"
I20260926 05:45:47.930929 13130 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:47.931835 13145 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:47.932021 13130 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:47.932152 13130 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "9a6105776d434995b5bd2796df70c534"
format_stamp: "Formatted at 2026-09-26 05:45:47 on dist-test-slave-1fdt"
server_key: "c0e45ccf40f8dbff48e5194fabcd897e"
server_key_iv: "8fd14344f21ef93d9a7bb7b1a1476c28"
server_key_version: "encryptionkey@0"
I20260926 05:45:47.932225 13130 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:47.941042 13130 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:47.943378 13130 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:47.943537 13130 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:47.947443 13130 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35849
I20260926 05:45:47.947474 13197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35849 every 8 connection(s)
I20260926 05:45:47.947813 13130 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:47.948626 13198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:47.948861 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13130
I20260926 05:45:47.948989 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:47.949190 24407 external_mini_cluster.cc:1428] Setting key eace76e56ad2f1d562cf336581e7a354
I20260926 05:45:47.951140 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534: Bootstrap starting.
I20260926 05:45:47.951716 13198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:47.952009 13198 log.cc:824] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:47.952667 13198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534: No bootstrap required, opened a new log
I20260926 05:45:47.954512 13198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6105776d434995b5bd2796df70c534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35849 } }
I20260926 05:45:47.954664 13198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:47.954741 13198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a6105776d434995b5bd2796df70c534, State: Initialized, Role: FOLLOWER
I20260926 05:45:47.954916 13198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6105776d434995b5bd2796df70c534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35849 } }
I20260926 05:45:47.955020 13198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:47.955051 13198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:47.955099 13198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 05:45:47 dist-test-slave-1fdt krb5kdc[13117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401547, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 05:45:47.955801 13198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6105776d434995b5bd2796df70c534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35849 } }
I20260926 05:45:47.955912 13198 leader_election.cc:302] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a6105776d434995b5bd2796df70c534; no voters:
I20260926 05:45:47.956076 13198 leader_election.cc:288] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:47.956135 13203 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:47.956279 13203 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [term 1 LEADER]: Becoming Leader. State: Replica: 9a6105776d434995b5bd2796df70c534, State: Running, Role: LEADER
I20260926 05:45:47.956338 13198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:47.956364 13203 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6105776d434995b5bd2796df70c534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35849 } }
I20260926 05:45:47.956753 13203 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a6105776d434995b5bd2796df70c534. Latest consensus state: current_term: 1 leader_uuid: "9a6105776d434995b5bd2796df70c534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6105776d434995b5bd2796df70c534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35849 } } }
I20260926 05:45:47.956836 13203 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:47.957000 13204 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a6105776d434995b5bd2796df70c534" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a6105776d434995b5bd2796df70c534" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35849 } } }
I20260926 05:45:47.957067 13204 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:47.957187 13210 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:47.957676 13210 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:47.959506 13210 catalog_manager.cc:1347] Generated new cluster ID: 9ce7b834ca1441619ba7a7d610155fdf
I20260926 05:45:47.959559 13210 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:47.960214 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:47.949722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:36939 (local address 127.0.0.1:35849)
0926 05:45:47.949928 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:47.949936 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:47.949963 (+ 27us) server_negotiation.cc:407] Connection header received
0926 05:45:47.950360 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:47.950366 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:47.950412 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:47.950564 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:47.951170 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:47.952209 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:47.953016 (+ 807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:47.953436 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:47.953562 (+ 126us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:47.956273 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:47.956277 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:47.956277 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:47.956288 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:47.958501 (+ 2213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:47.958946 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:47.958949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:47.958950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:47.959064 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:47.959323 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:47.959324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:47.959325 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:47.959647 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:47.959824 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:47.959950 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 05:45:47.960053 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":71,"threads_started":1}
I20260926 05:45:47.961151 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:47.972075 13201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:47.962554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:51399 (local address 127.0.0.1:35849)
0926 05:45:47.962591 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:47.962592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:47.965980 (+ 3388us) server_negotiation.cc:407] Connection header received
0926 05:45:47.966058 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:47.966059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:47.966070 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:47.966142 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:47.966685 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:47.967175 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:47.967876 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:47.968131 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:47.968209 (+ 78us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:47.969435 (+ 1226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:47.969436 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:47.969437 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:47.969443 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:47.970886 (+ 1443us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:47.971271 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:47.971272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:47.971273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:47.971331 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:47.971560 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:47.971562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:47.971562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:47.971672 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:47.971783 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:47.971901 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 05:45:47.971965 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:47.990674 13210 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:47.991271 13210 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:47.995433 13210 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a6105776d434995b5bd2796df70c534: Generated new TSK 0
I20260926 05:45:47.995692 13210 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:47.999635 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13130
W20260926 05:45:48.003690 13115 connection.cc:535] client connection to 127.0.0.1:35849 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35849): Connection reset by peer (error 104)
2026-09-26T05:45:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:48 dist-test-slave-1fdt krb5kdc[13229](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:48 dist-test-slave-1fdt krb5kdc[13229](info): set up 2 sockets
Sep 26 05:45:48 dist-test-slave-1fdt krb5kdc[13229](info): commencing operation
krb5kdc: starting...
W20260926 05:45:49.033972 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.017s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:49 dist-test-slave-1fdt krb5kdc[13229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401549, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:49.164079 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45851
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45851
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:49.243971 13242 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:49.244237 13242 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:49.244279 13242 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:49.245455 13242 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:49.245532 13242 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:49.245556 13242 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:49.245586 13242 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:49.245617 13242 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:49.247107 13242 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45851
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13242
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:49.247406 13242 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:49.247638 13242 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:49.252525 13250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:49.252560 13247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:49.252539 13248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:49.252872 13242 server_base.cc:1034] running on GCE node
I20260926 05:45:49.253129 13242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:49.253413 13242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:49.254587 13242 hybrid_clock.cc:648] HybridClock initialized: now 1790401549254620 us; error 112 us; skew 500 ppm
Sep 26 05:45:49 dist-test-slave-1fdt krb5kdc[13229](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401549, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:49.257072 13242 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:49.257545 13242 webserver.cc:460] Webserver started at http://127.0.0.1:38495/ using document root <none> and password file <none>
I20260926 05:45:49.257731 13242 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:49.257835 13242 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:49.257929 13242 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:49.258944 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "461500db4db341329013e889085f74ae"
format_stamp: "Formatted at 2026-09-26 05:45:49 on dist-test-slave-1fdt"
server_key: "807265a599aa1d36fa0a368eb9a104df"
server_key_iv: "c1f5f7fd401ac3fb3183e6481bab0c34"
server_key_version: "encryptionkey@0"
I20260926 05:45:49.259284 13242 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "461500db4db341329013e889085f74ae"
format_stamp: "Formatted at 2026-09-26 05:45:49 on dist-test-slave-1fdt"
server_key: "807265a599aa1d36fa0a368eb9a104df"
server_key_iv: "c1f5f7fd401ac3fb3183e6481bab0c34"
server_key_version: "encryptionkey@0"
I20260926 05:45:49.260547 13242 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:49.261282 13257 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:49.261490 13242 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 05:45:49.261596 13242 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "461500db4db341329013e889085f74ae"
format_stamp: "Formatted at 2026-09-26 05:45:49 on dist-test-slave-1fdt"
server_key: "807265a599aa1d36fa0a368eb9a104df"
server_key_iv: "c1f5f7fd401ac3fb3183e6481bab0c34"
server_key_version: "encryptionkey@0"
I20260926 05:45:49.261693 13242 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:49.282065 13242 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:49.284444 13242 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:49.284617 13242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:49.288522 13242 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45851
I20260926 05:45:49.288590 13309 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45851 every 8 connection(s)
I20260926 05:45:49.288962 13242 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:49.289840 13310 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:49.292078 13310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae: Bootstrap starting.
I20260926 05:45:49.292658 13310 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:49.292946 13310 log.cc:824] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:49.293617 13310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae: No bootstrap required, opened a new log
I20260926 05:45:49.295222 13310 raft_consensus.cc:348] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461500db4db341329013e889085f74ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45851 } }
I20260926 05:45:49.295356 13310 raft_consensus.cc:374] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:49.295405 13310 raft_consensus.cc:729] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 461500db4db341329013e889085f74ae, State: Initialized, Role: FOLLOWER
I20260926 05:45:49.295555 13310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461500db4db341329013e889085f74ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45851 } }
I20260926 05:45:49.295615 13310 raft_consensus.cc:388] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:49.295635 13310 raft_consensus.cc:482] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:49.295712 13310 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:49.296242 13310 raft_consensus.cc:504] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461500db4db341329013e889085f74ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45851 } }
I20260926 05:45:49.296353 13310 leader_election.cc:302] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 461500db4db341329013e889085f74ae; no voters:
I20260926 05:45:49.296532 13310 leader_election.cc:288] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:49.296566 13313 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:49.296736 13313 raft_consensus.cc:686] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [term 1 LEADER]: Becoming Leader. State: Replica: 461500db4db341329013e889085f74ae, State: Running, Role: LEADER
I20260926 05:45:49.296797 13310 sys_catalog.cc:564] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:49.296880 13313 consensus_queue.cc:237] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461500db4db341329013e889085f74ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45851 } }
I20260926 05:45:49.297452 13314 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "461500db4db341329013e889085f74ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461500db4db341329013e889085f74ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45851 } } }
I20260926 05:45:49.297539 13314 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:49.297925 13322 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:49.298064 13315 sys_catalog.cc:455] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [sys.catalog]: SysCatalogTable state changed. Reason: New leader 461500db4db341329013e889085f74ae. Latest consensus state: current_term: 1 leader_uuid: "461500db4db341329013e889085f74ae" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "461500db4db341329013e889085f74ae" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45851 } } }
I20260926 05:45:49.298178 13315 sys_catalog.cc:458] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:49.298484 13322 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:49.298882 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13242
I20260926 05:45:49.298997 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:49.299190 24407 external_mini_cluster.cc:1428] Setting key aa584f8fb380371cd0201ca4938b2ef5
I20260926 05:45:49.300702 13322 catalog_manager.cc:1347] Generated new cluster ID: d7990ed5a4024b79ad3ed430c9e9311c
I20260926 05:45:49.300756 13322 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:49.305147 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:49.299652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:47387 (local address 127.0.0.1:45851)
0926 05:45:49.299825 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:49.299833 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:49.299864 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:45:49.299980 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:49.299986 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:49.300045 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:49.300212 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:49.300391 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:49.301736 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:49.302921 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:49.303532 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:49.303626 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:49.304384 (+ 758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:49.304388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:49.304389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:49.304592 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:49.304630 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:49.304644 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:49.304722 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:49.304839 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 05:45:49.304953 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260926 05:45:49.305974 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:49.310999 13333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:49.307478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:41601 (local address 127.0.0.1:45851)
0926 05:45:49.307508 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:49.307510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:49.307661 (+ 151us) server_negotiation.cc:407] Connection header received
0926 05:45:49.307722 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:49.307723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:49.307733 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:49.307786 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:49.307902 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:49.308652 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:49.309676 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:49.310016 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:49.310072 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:49.310750 (+ 678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:49.310752 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:49.310752 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:49.310756 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:49.310770 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:49.310775 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:49.310829 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:49.310918 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 05:45:49.310929 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260926 05:45:49.316046 13322 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:49.316730 13322 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:49.323634 13322 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 461500db4db341329013e889085f74ae: Generated new TSK 0
I20260926 05:45:49.323807 13322 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:49.327854 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13242
W20260926 05:45:49.332312 13227 connection.cc:535] client connection to 127.0.0.1:45851 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45851): Connection reset by peer (error 104)
2026-09-26T05:45:49Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1328 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:49 dist-test-slave-1fdt krb5kdc[13341](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:49 dist-test-slave-1fdt krb5kdc[13341](info): set up 2 sockets
Sep 26 05:45:49 dist-test-slave-1fdt krb5kdc[13341](info): commencing operation
krb5kdc: starting...
W20260926 05:45:50.381953 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.035s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:50 dist-test-slave-1fdt krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401550, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 05:45:50.500890 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:50.585377 13354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:50.585610 13354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:50.585664 13354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:50.586862 13354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:50.586930 13354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:50.586957 13354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:50.586987 13354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:50.587008 13354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:50.588485 13354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13354
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:50.588757 13354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:50.588984 13354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:50.593951 13360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:50.593930 13359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:50.593951 13362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:50.594184 13354 server_base.cc:1034] running on GCE node
I20260926 05:45:50.594409 13354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:50.594663 13354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:50.595805 13354 hybrid_clock.cc:648] HybridClock initialized: now 1790401550595774 us; error 46 us; skew 500 ppm
Sep 26 05:45:50 dist-test-slave-1fdt krb5kdc[13341](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401550, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:50.598286 13354 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 05:45:50.598745 13354 webserver.cc:460] Webserver started at http://127.0.0.1:41917/ using document root <none> and password file <none>
I20260926 05:45:50.598948 13354 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:50.598987 13354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:50.599074 13354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:50.600087 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "7746f99219704914b823c7d64f8ac48f"
format_stamp: "Formatted at 2026-09-26 05:45:50 on dist-test-slave-1fdt"
server_key: "ff18b3576c9a9ce10c6e243c2ec0a00d"
server_key_iv: "e7593089137d43c022b831678fce7003"
server_key_version: "encryptionkey@0"
I20260926 05:45:50.600382 13354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "7746f99219704914b823c7d64f8ac48f"
format_stamp: "Formatted at 2026-09-26 05:45:50 on dist-test-slave-1fdt"
server_key: "ff18b3576c9a9ce10c6e243c2ec0a00d"
server_key_iv: "e7593089137d43c022b831678fce7003"
server_key_version: "encryptionkey@0"
I20260926 05:45:50.601624 13354 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:45:50.602402 13369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:50.602586 13354 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:50.602679 13354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "7746f99219704914b823c7d64f8ac48f"
format_stamp: "Formatted at 2026-09-26 05:45:50 on dist-test-slave-1fdt"
server_key: "ff18b3576c9a9ce10c6e243c2ec0a00d"
server_key_iv: "e7593089137d43c022b831678fce7003"
server_key_version: "encryptionkey@0"
I20260926 05:45:50.602757 13354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:50.606109 13354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:50.607944 13354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:50.608093 13354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:50.611941 13354 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32931
I20260926 05:45:50.612020 13421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32931 every 8 connection(s)
I20260926 05:45:50.612347 13354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:50.613327 13422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:50.615698 13422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f: Bootstrap starting.
I20260926 05:45:50.615711 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13354
I20260926 05:45:50.615818 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:50.615990 24407 external_mini_cluster.cc:1428] Setting key d532997d46b0b6cb26440e1604ea8a27
I20260926 05:45:50.616420 13422 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:50.616802 13422 log.cc:824] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:50.617561 13422 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f: No bootstrap required, opened a new log
I20260926 05:45:50.618558 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:50.616511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:46995 (local address 127.0.0.1:32931)
0926 05:45:50.616702 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:50.616709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:50.616733 (+ 24us) server_negotiation.cc:407] Connection header received
0926 05:45:50.617227 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:50.617233 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:50.617260 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:50.617379 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:50.618120 (+ 741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:50.618123 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:50.618123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:50.618127 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:50.618164 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:50.618174 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:50.618249 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:50.618305 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 05:45:50.618365 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":77,"threads_started":1}
I20260926 05:45:50.619510 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:50.619985 13422 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7746f99219704914b823c7d64f8ac48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32931 } }
I20260926 05:45:50.620177 13422 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:50.620210 13422 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7746f99219704914b823c7d64f8ac48f, State: Initialized, Role: FOLLOWER
I20260926 05:45:50.620340 13422 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7746f99219704914b823c7d64f8ac48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32931 } }
I20260926 05:45:50.620435 13422 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:50.620464 13422 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:50.620512 13422 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:50.621280 13422 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7746f99219704914b823c7d64f8ac48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32931 } }
I20260926 05:45:50.621398 13422 leader_election.cc:302] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7746f99219704914b823c7d64f8ac48f; no voters:
I20260926 05:45:50.621567 13422 leader_election.cc:288] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:50.621733 13433 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:50.621871 13422 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:50.621969 13433 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [term 1 LEADER]: Becoming Leader. State: Replica: 7746f99219704914b823c7d64f8ac48f, State: Running, Role: LEADER
I20260926 05:45:50.622124 13433 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7746f99219704914b823c7d64f8ac48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32931 } }
I20260926 05:45:50.622346 13425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:50.621113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.149:32995 (local address 127.0.0.1:32931)
0926 05:45:50.621149 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:50.621151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:50.621211 (+ 60us) server_negotiation.cc:407] Connection header received
0926 05:45:50.621278 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:50.621279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:50.621282 (+ 3us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:50.621325 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:50.621980 (+ 655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:50.621981 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:50.621981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:50.621986 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:50.622002 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:50.622006 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:50.622061 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:50.622260 (+ 199us) server_negotiation.cc:299] Negotiation successful
0926 05:45:50.622294 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:50.622548 13433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7746f99219704914b823c7d64f8ac48f. Latest consensus state: current_term: 1 leader_uuid: "7746f99219704914b823c7d64f8ac48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7746f99219704914b823c7d64f8ac48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32931 } } }
I20260926 05:45:50.622623 13433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:50.622560 13432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7746f99219704914b823c7d64f8ac48f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7746f99219704914b823c7d64f8ac48f" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32931 } } }
I20260926 05:45:50.622663 13432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f [sys.catalog]: This master's current role is: LEADER
W20260926 05:45:50.623772 13447 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:45:50.623842 13447 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:45:50.623914 13441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:50.624111 13441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:50.625469 13441 catalog_manager.cc:1347] Generated new cluster ID: 6d2587f734404f32a4fa75b2c34db02a
I20260926 05:45:50.625519 13441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:50.629920 13441 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:50.630087 13441 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:50.634096 13441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7746f99219704914b823c7d64f8ac48f: Generated new TSK 0
I20260926 05:45:50.634291 13441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:50.642071 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13354
2026-09-26T05:45:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1313 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:50 dist-test-slave-1fdt krb5kdc[13453](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:50 dist-test-slave-1fdt krb5kdc[13453](info): set up 2 sockets
Sep 26 05:45:50 dist-test-slave-1fdt krb5kdc[13453](info): commencing operation
krb5kdc: starting...
W20260926 05:45:51.673769 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:51 dist-test-slave-1fdt krb5kdc[13453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:51.792984 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41655
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:34947
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:41655
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:51.873353 13466 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:51.873569 13466 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:51.873591 13466 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:51.874821 13466 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:51.874866 13466 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:51.874879 13466 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:51.874892 13466 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:51.874904 13466 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:51.876390 13466 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:34947
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:41655
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:41655
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13466
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:51.876650 13466 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:51.876885 13466 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:51.881865 13472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:51.881875 13471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:51.881888 13474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:51.881906 13466 server_base.cc:1034] running on GCE node
I20260926 05:45:51.882380 13466 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:51.882694 13466 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:51.883883 13466 hybrid_clock.cc:648] HybridClock initialized: now 1790401551883854 us; error 41 us; skew 500 ppm
Sep 26 05:45:51 dist-test-slave-1fdt krb5kdc[13453](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401551, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:51.889928 13466 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:51.890337 13466 webserver.cc:460] Webserver started at http://127.23.213.254:43257/ using document root <none> and password file <none>
I20260926 05:45:51.890571 13466 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:51.890627 13466 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:51.890760 13466 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:51.891851 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "6fde94b86536416093a22ccd78be43db"
format_stamp: "Formatted at 2026-09-26 05:45:51 on dist-test-slave-1fdt"
server_key: "b12347a42aa14191e9d44770f0a07e2f"
server_key_iv: "8efcc389a5b8ed7008cf6848ebe4090e"
server_key_version: "encryptionkey@0"
I20260926 05:45:51.892217 13466 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "6fde94b86536416093a22ccd78be43db"
format_stamp: "Formatted at 2026-09-26 05:45:51 on dist-test-slave-1fdt"
server_key: "b12347a42aa14191e9d44770f0a07e2f"
server_key_iv: "8efcc389a5b8ed7008cf6848ebe4090e"
server_key_version: "encryptionkey@0"
I20260926 05:45:51.893631 13466 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:51.894412 13481 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:51.894613 13466 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:51.894707 13466 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "6fde94b86536416093a22ccd78be43db"
format_stamp: "Formatted at 2026-09-26 05:45:51 on dist-test-slave-1fdt"
server_key: "b12347a42aa14191e9d44770f0a07e2f"
server_key_iv: "8efcc389a5b8ed7008cf6848ebe4090e"
server_key_version: "encryptionkey@0"
I20260926 05:45:51.894784 13466 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:51.923560 13466 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:51.925570 13466 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:51.925779 13466 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:51.929962 13466 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:41655
I20260926 05:45:51.930001 13533 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:41655 every 8 connection(s)
I20260926 05:45:51.930357 13466 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:51.931160 13534 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:51.933454 13534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db: Bootstrap starting.
I20260926 05:45:51.934129 13534 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:51.934417 13534 log.cc:824] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:51.935164 13534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db: No bootstrap required, opened a new log
I20260926 05:45:51.936726 13534 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fde94b86536416093a22ccd78be43db" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41655 } }
I20260926 05:45:51.936858 13534 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:51.936880 13534 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fde94b86536416093a22ccd78be43db, State: Initialized, Role: FOLLOWER
I20260926 05:45:51.937042 13534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fde94b86536416093a22ccd78be43db" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41655 } }
I20260926 05:45:51.937104 13534 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:51.937126 13534 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:51.937196 13534 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:51.938060 13534 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fde94b86536416093a22ccd78be43db" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41655 } }
I20260926 05:45:51.938088 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13466
I20260926 05:45:51.938175 13534 leader_election.cc:302] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fde94b86536416093a22ccd78be43db; no voters:
I20260926 05:45:51.938211 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:51.938400 13534 leader_election.cc:288] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:51.938421 24407 external_mini_cluster.cc:1428] Setting key 9b096d8e008b6bbbc3fe6d5ada8a5405
I20260926 05:45:51.938418 13537 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:51.938532 13537 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [term 1 LEADER]: Becoming Leader. State: Replica: 6fde94b86536416093a22ccd78be43db, State: Running, Role: LEADER
I20260926 05:45:51.938660 13534 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:51.938671 13537 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fde94b86536416093a22ccd78be43db" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41655 } }
I20260926 05:45:51.939169 13538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fde94b86536416093a22ccd78be43db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fde94b86536416093a22ccd78be43db" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41655 } } }
I20260926 05:45:51.939193 13539 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fde94b86536416093a22ccd78be43db. Latest consensus state: current_term: 1 leader_uuid: "6fde94b86536416093a22ccd78be43db" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fde94b86536416093a22ccd78be43db" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 41655 } } }
I20260926 05:45:51.939340 13538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:51.939433 13539 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:51.939720 13548 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:51.940325 13548 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:51.942019 13548 catalog_manager.cc:1347] Generated new cluster ID: 8311bf27ebb94381aa4f2776cb9fb2f5
I20260926 05:45:51.942070 13548 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:45:51 dist-test-slave-1fdt krb5kdc[13453](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401551, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:51.951455 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:51.939077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34735 (local address 127.23.213.254:41655)
0926 05:45:51.939700 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:51.939709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:51.939747 (+ 38us) server_negotiation.cc:407] Connection header received
0926 05:45:51.939868 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:51.939875 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:51.940210 (+ 335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:51.940388 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:51.941057 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:51.942428 (+ 1371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:51.943558 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:51.944150 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:51.944247 (+ 97us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:51.947190 (+ 2943us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:51.947203 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:51.947203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:51.947217 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:51.949493 (+ 2276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:51.949974 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:51.949977 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:51.949978 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:51.950123 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:51.950372 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:51.950373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:51.950373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:51.950696 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:51.950900 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:51.951001 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 05:45:51.951248 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":82,"threads_started":1}
I20260926 05:45:51.952548 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:51.953892 13548 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:51.954422 13548 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:51.961997 13543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:51.954099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37263 (local address 127.23.213.254:41655)
0926 05:45:51.954135 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:51.954137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:51.954265 (+ 128us) server_negotiation.cc:407] Connection header received
0926 05:45:51.954365 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:51.954366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:51.954427 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:51.954476 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:51.955219 (+ 743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:51.955970 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:51.957055 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:51.957404 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:51.957447 (+ 43us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:51.959018 (+ 1571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:51.959019 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:51.959019 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:51.959026 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:51.960564 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:51.961015 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:51.961016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:51.961016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:51.961070 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:51.961315 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:51.961315 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:51.961316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:51.961430 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:51.961507 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:51.961810 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 05:45:51.961879 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:45:51.963806 13548 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fde94b86536416093a22ccd78be43db: Generated new TSK 0
I20260926 05:45:51.963994 13548 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:51.966770 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13466
W20260926 05:45:51.972869 13451 connection.cc:535] client connection to 127.23.213.254:41655 recv error: Network error: failed to read from TLS socket (remote: 127.23.213.254:41655): Connection reset by peer (error 104)
2026-09-26T05:45:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1326 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:52 dist-test-slave-1fdt krb5kdc[13565](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:52 dist-test-slave-1fdt krb5kdc[13565](info): set up 2 sockets
Sep 26 05:45:52 dist-test-slave-1fdt krb5kdc[13565](info): commencing operation
krb5kdc: starting...
W20260926 05:45:53.006114 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:53 dist-test-slave-1fdt krb5kdc[13565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401553, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:53.122681 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38121
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:37133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38121
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:53.202557 13578 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:53.202777 13578 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:53.202800 13578 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:53.203876 13578 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:53.203908 13578 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:53.203922 13578 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:53.203934 13578 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:53.203946 13578 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:53.205307 13578 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:37133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38121
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38121
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13578
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:53.205484 13578 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:53.205643 13578 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:53.210397 13586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:53.210388 13583 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:53.210418 13584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:53.210737 13578 server_base.cc:1034] running on GCE node
I20260926 05:45:53.210988 13578 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:53.211273 13578 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:53.212414 13578 hybrid_clock.cc:648] HybridClock initialized: now 1790401553212392 us; error 37 us; skew 500 ppm
Sep 26 05:45:53 dist-test-slave-1fdt krb5kdc[13565](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401553, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:53.215029 13578 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:53.215479 13578 webserver.cc:460] Webserver started at http://127.23.213.254:37723/ using document root <none> and password file <none>
I20260926 05:45:53.215721 13578 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:53.215768 13578 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:53.215883 13578 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:53.216919 13578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "7cb46d5bbd96413b8807f269d09ff10a"
format_stamp: "Formatted at 2026-09-26 05:45:53 on dist-test-slave-1fdt"
server_key: "46835749bb97ab4b826e2639f1b23665"
server_key_iv: "6eea61841b93e33a65e44784a509f5e3"
server_key_version: "encryptionkey@0"
I20260926 05:45:53.217274 13578 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "7cb46d5bbd96413b8807f269d09ff10a"
format_stamp: "Formatted at 2026-09-26 05:45:53 on dist-test-slave-1fdt"
server_key: "46835749bb97ab4b826e2639f1b23665"
server_key_iv: "6eea61841b93e33a65e44784a509f5e3"
server_key_version: "encryptionkey@0"
I20260926 05:45:53.218640 13578 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:53.219341 13593 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:53.219502 13578 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:53.219594 13578 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "7cb46d5bbd96413b8807f269d09ff10a"
format_stamp: "Formatted at 2026-09-26 05:45:53 on dist-test-slave-1fdt"
server_key: "46835749bb97ab4b826e2639f1b23665"
server_key_iv: "6eea61841b93e33a65e44784a509f5e3"
server_key_version: "encryptionkey@0"
I20260926 05:45:53.219673 13578 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:53.234422 13578 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:53.236791 13578 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:53.236956 13578 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:53.241055 13578 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38121
I20260926 05:45:53.241098 13645 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38121 every 8 connection(s)
I20260926 05:45:53.241472 13578 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:53.242344 13646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:53.244530 13646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a: Bootstrap starting.
I20260926 05:45:53.245064 13646 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:53.245293 13646 log.cc:824] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:53.245942 13646 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a: No bootstrap required, opened a new log
I20260926 05:45:53.247433 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13578
I20260926 05:45:53.247561 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:53.247730 24407 external_mini_cluster.cc:1428] Setting key 6ca97d6391bd8161a8440c13db981c4f
I20260926 05:45:53.247756 13646 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb46d5bbd96413b8807f269d09ff10a" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38121 } }
I20260926 05:45:53.247937 13646 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:53.247964 13646 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7cb46d5bbd96413b8807f269d09ff10a, State: Initialized, Role: FOLLOWER
I20260926 05:45:53.248131 13646 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb46d5bbd96413b8807f269d09ff10a" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38121 } }
I20260926 05:45:53.248214 13646 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:53.248252 13646 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:53.248353 13646 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:53.249014 13646 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb46d5bbd96413b8807f269d09ff10a" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38121 } }
I20260926 05:45:53.249107 13646 leader_election.cc:302] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7cb46d5bbd96413b8807f269d09ff10a; no voters:
I20260926 05:45:53.249298 13646 leader_election.cc:288] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:53.249323 13651 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:53.249512 13651 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [term 1 LEADER]: Becoming Leader. State: Replica: 7cb46d5bbd96413b8807f269d09ff10a, State: Running, Role: LEADER
I20260926 05:45:53.249554 13646 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:53.249635 13651 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb46d5bbd96413b8807f269d09ff10a" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38121 } }
I20260926 05:45:53.249970 13651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7cb46d5bbd96413b8807f269d09ff10a. Latest consensus state: current_term: 1 leader_uuid: "7cb46d5bbd96413b8807f269d09ff10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb46d5bbd96413b8807f269d09ff10a" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38121 } } }
I20260926 05:45:53.250063 13651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:53.249981 13652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7cb46d5bbd96413b8807f269d09ff10a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7cb46d5bbd96413b8807f269d09ff10a" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38121 } } }
I20260926 05:45:53.250187 13652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:53.250433 13658 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:53.250986 13658 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:53.252727 13668 mvcc.cc:205] Tried to move back new op lower bound from 7333484762120581120 to 7333484762110980096. Current Snapshot: MvccSnapshot[applied={T|T < 7333484762120581120}]
I20260926 05:45:53.253628 13658 catalog_manager.cc:1347] Generated new cluster ID: a98bca769ca841d9bbb87e4a0493abdd
I20260926 05:45:53.253682 13658 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:53.254127 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:53.248188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52507 (local address 127.23.213.254:38121)
0926 05:45:53.248385 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:53.248395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:53.248591 (+ 196us) server_negotiation.cc:407] Connection header received
0926 05:45:53.248965 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:53.248971 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:53.249016 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:53.249173 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:53.249471 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:53.251023 (+ 1552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:53.252093 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:53.252620 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:53.252750 (+ 130us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:53.253387 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:53.253391 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:53.253391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:53.253607 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:53.253638 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:53.253649 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:53.253767 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:53.253894 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 05:45:53.253923 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":93,"threads_started":1}
I20260926 05:45:53.255043 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:53.260876 13650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:53.256368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60521 (local address 127.23.213.254:38121)
0926 05:45:53.256404 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:53.256406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:53.257937 (+ 1531us) server_negotiation.cc:407] Connection header received
0926 05:45:53.258010 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:53.258011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:53.258022 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:53.258071 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:53.258193 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:53.258924 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:53.259640 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:53.259869 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:53.259940 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:53.260646 (+ 706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:53.260648 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:53.260648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:53.260652 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:53.260667 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:53.260672 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:53.260726 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:53.260786 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 05:45:53.260797 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:53.278187 13658 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:53.278753 13658 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:53.288935 13658 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7cb46d5bbd96413b8807f269d09ff10a: Generated new TSK 0
I20260926 05:45:53.289144 13658 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:53.298624 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13578
W20260926 05:45:53.302791 13563 connection.cc:535] client connection to 127.23.213.254:38121 recv error: Network error: failed to read from TLS socket (remote: 127.23.213.254:38121): Connection reset by peer (error 104)
2026-09-26T05:45:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1329 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:53 dist-test-slave-1fdt krb5kdc[13677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:53 dist-test-slave-1fdt krb5kdc[13677](info): set up 2 sockets
Sep 26 05:45:53 dist-test-slave-1fdt krb5kdc[13677](info): commencing operation
krb5kdc: starting...
W20260926 05:45:54.325822 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.010s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:54 dist-test-slave-1fdt krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401554, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:54.444543 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43661
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:44185
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:43661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:54.525067 13690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:54.525309 13690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:54.525346 13690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:54.526566 13690 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:54.526635 13690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:54.526660 13690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:54.526681 13690 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:54.526711 13690 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:54.528172 13690 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:44185
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:43661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:43661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13690
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:54.528450 13690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:54.528683 13690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:54.533538 13698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:54.533581 13690 server_base.cc:1034] running on GCE node
W20260926 05:45:54.533499 13695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:54.533524 13696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:54.533967 13690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:54.534292 13690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:54.535452 13690 hybrid_clock.cc:648] HybridClock initialized: now 1790401554535445 us; error 50 us; skew 500 ppm
Sep 26 05:45:54 dist-test-slave-1fdt krb5kdc[13677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401554, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:54.537914 13690 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:54.538359 13690 webserver.cc:460] Webserver started at http://127.23.213.254:37679/ using document root <none> and password file <none>
I20260926 05:45:54.538585 13690 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:54.538651 13690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:54.538797 13690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:54.539841 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "31372f9edcd24afe8e03fc5ec6660598"
format_stamp: "Formatted at 2026-09-26 05:45:54 on dist-test-slave-1fdt"
server_key: "922283711ed294db8595764cef429ccb"
server_key_iv: "56fc329659a410c11f5266e61d2ae843"
server_key_version: "encryptionkey@0"
I20260926 05:45:54.540206 13690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "31372f9edcd24afe8e03fc5ec6660598"
format_stamp: "Formatted at 2026-09-26 05:45:54 on dist-test-slave-1fdt"
server_key: "922283711ed294db8595764cef429ccb"
server_key_iv: "56fc329659a410c11f5266e61d2ae843"
server_key_version: "encryptionkey@0"
I20260926 05:45:54.541510 13690 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 05:45:54.542315 13705 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:54.542490 13690 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:54.542588 13690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "31372f9edcd24afe8e03fc5ec6660598"
format_stamp: "Formatted at 2026-09-26 05:45:54 on dist-test-slave-1fdt"
server_key: "922283711ed294db8595764cef429ccb"
server_key_iv: "56fc329659a410c11f5266e61d2ae843"
server_key_version: "encryptionkey@0"
I20260926 05:45:54.542687 13690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:54.545953 13690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:54.547794 13690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:54.547951 13690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:54.551982 13690 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:43661
I20260926 05:45:54.552026 13757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:43661 every 8 connection(s)
I20260926 05:45:54.552345 13690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:54.553278 13758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:54.555505 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598: Bootstrap starting.
I20260926 05:45:54.556020 13758 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:54.556252 13758 log.cc:824] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:54.556898 13758 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598: No bootstrap required, opened a new log
I20260926 05:45:54.558513 13758 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31372f9edcd24afe8e03fc5ec6660598" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43661 } }
I20260926 05:45:54.558657 13758 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:54.558681 13758 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31372f9edcd24afe8e03fc5ec6660598, State: Initialized, Role: FOLLOWER
I20260926 05:45:54.558779 13758 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31372f9edcd24afe8e03fc5ec6660598" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43661 } }
I20260926 05:45:54.558830 13758 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:54.558849 13758 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:54.558876 13758 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:54.559091 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13690
I20260926 05:45:54.559209 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:54.559402 24407 external_mini_cluster.cc:1428] Setting key b808a95b34f8bef1afbf5c66c568b6e1
I20260926 05:45:54.559863 13758 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31372f9edcd24afe8e03fc5ec6660598" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43661 } }
I20260926 05:45:54.559974 13758 leader_election.cc:302] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31372f9edcd24afe8e03fc5ec6660598; no voters:
I20260926 05:45:54.560139 13758 leader_election.cc:288] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:54.560175 13763 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:54.560343 13763 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [term 1 LEADER]: Becoming Leader. State: Replica: 31372f9edcd24afe8e03fc5ec6660598, State: Running, Role: LEADER
I20260926 05:45:54.560385 13758 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:54.560505 13763 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31372f9edcd24afe8e03fc5ec6660598" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43661 } }
I20260926 05:45:54.561009 13763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31372f9edcd24afe8e03fc5ec6660598. Latest consensus state: current_term: 1 leader_uuid: "31372f9edcd24afe8e03fc5ec6660598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31372f9edcd24afe8e03fc5ec6660598" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43661 } } }
I20260926 05:45:54.561158 13763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:54.561839 13772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:54.561873 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:54.559842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57925 (local address 127.23.213.254:43661)
0926 05:45:54.560101 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:54.560116 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:54.560172 (+ 56us) server_negotiation.cc:407] Connection header received
0926 05:45:54.560290 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:54.560298 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:54.560331 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:54.560475 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:54.561293 (+ 818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:54.561298 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:54.561298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:54.561304 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:54.561352 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:54.561366 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:54.561431 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:54.561611 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 05:45:54.561668 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203,"thread_start_us":110,"threads_started":1}
I20260926 05:45:54.561021 13764 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31372f9edcd24afe8e03fc5ec6660598" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31372f9edcd24afe8e03fc5ec6660598" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 43661 } } }
I20260926 05:45:54.562347 13764 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:54.562371 13772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:54.562906 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:54.563810 13772 catalog_manager.cc:1347] Generated new cluster ID: 96503814edc7433b950561b99c7e6ca8
I20260926 05:45:54.563849 13772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:54.565420 13762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:54.564385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48279 (local address 127.23.213.254:43661)
0926 05:45:54.564419 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:54.564421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:54.564546 (+ 125us) server_negotiation.cc:407] Connection header received
0926 05:45:54.564574 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:54.564575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:54.564580 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:54.564648 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:54.565200 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:54.565201 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:54.565202 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:54.565206 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:54.565222 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:54.565227 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:54.565292 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:54.565352 (+ 60us) server_negotiation.cc:299] Negotiation successful
0926 05:45:54.565364 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:45:54.571033 13772 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:54.571184 13772 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:54.577086 13772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31372f9edcd24afe8e03fc5ec6660598: Generated new TSK 0
I20260926 05:45:54.577258 13772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:54.580796 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13690
2026-09-26T05:45:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1282 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:54 dist-test-slave-1fdt krb5kdc[13789](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:54 dist-test-slave-1fdt krb5kdc[13789](info): set up 2 sockets
Sep 26 05:45:54 dist-test-slave-1fdt krb5kdc[13789](info): commencing operation
krb5kdc: starting...
W20260926 05:45:55.638033 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:55 dist-test-slave-1fdt krb5kdc[13789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:55.756208 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:45025
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:34921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:45025
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:55.837507 13802 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:55.837726 13802 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:55.837773 13802 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:55.838905 13802 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:55.838940 13802 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:55.838954 13802 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:55.838967 13802 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:55.838980 13802 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:55.840368 13802 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:34921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:45025
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:45025
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13802
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:55.840602 13802 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:55.840842 13802 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:55.845948 13807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:55.845956 13810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:55.845986 13802 server_base.cc:1034] running on GCE node
W20260926 05:45:55.845989 13808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:55.846390 13802 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:55.846642 13802 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:55.847810 13802 hybrid_clock.cc:648] HybridClock initialized: now 1790401555847795 us; error 46 us; skew 500 ppm
Sep 26 05:45:55 dist-test-slave-1fdt krb5kdc[13789](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401555, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:55.850379 13802 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:55.850859 13802 webserver.cc:460] Webserver started at http://127.23.213.254:33637/ using document root <none> and password file <none>
I20260926 05:45:55.851082 13802 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:55.851148 13802 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:55.851307 13802 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:55.852344 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "c98d6f2af21b4fd19fc7ab8514206bd8"
format_stamp: "Formatted at 2026-09-26 05:45:55 on dist-test-slave-1fdt"
server_key: "dad1187730db553184a35c49ae187768"
server_key_iv: "f64cf2a5c2e6d09cda7f5e2996af2bcf"
server_key_version: "encryptionkey@0"
I20260926 05:45:55.852708 13802 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "c98d6f2af21b4fd19fc7ab8514206bd8"
format_stamp: "Formatted at 2026-09-26 05:45:55 on dist-test-slave-1fdt"
server_key: "dad1187730db553184a35c49ae187768"
server_key_iv: "f64cf2a5c2e6d09cda7f5e2996af2bcf"
server_key_version: "encryptionkey@0"
I20260926 05:45:55.854081 13802 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 05:45:55.854809 13817 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:55.855005 13802 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:55.855109 13802 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "c98d6f2af21b4fd19fc7ab8514206bd8"
format_stamp: "Formatted at 2026-09-26 05:45:55 on dist-test-slave-1fdt"
server_key: "dad1187730db553184a35c49ae187768"
server_key_iv: "f64cf2a5c2e6d09cda7f5e2996af2bcf"
server_key_version: "encryptionkey@0"
I20260926 05:45:55.855189 13802 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:55.879984 13802 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:55.882395 13802 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:55.882563 13802 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:55.886615 13802 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:45025
I20260926 05:45:55.886652 13869 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:45025 every 8 connection(s)
I20260926 05:45:55.887006 13802 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:55.887857 13870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:55.890254 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8: Bootstrap starting.
I20260926 05:45:55.890838 13870 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:55.891129 13870 log.cc:824] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:55.891422 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13802
I20260926 05:45:55.891551 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:55.891757 24407 external_mini_cluster.cc:1428] Setting key f0fb325d1af17f1bae89766384325d42
I20260926 05:45:55.891963 13870 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8: No bootstrap required, opened a new log
I20260926 05:45:55.893793 13870 raft_consensus.cc:348] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45025 } }
I20260926 05:45:55.893968 13870 raft_consensus.cc:374] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:55.894030 13870 raft_consensus.cc:729] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c98d6f2af21b4fd19fc7ab8514206bd8, State: Initialized, Role: FOLLOWER
I20260926 05:45:55.894161 13870 consensus_queue.cc:260] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45025 } }
I20260926 05:45:55.894261 13870 raft_consensus.cc:388] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:55.894306 13870 raft_consensus.cc:482] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:55.894354 13870 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:55.895203 13870 raft_consensus.cc:504] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45025 } }
I20260926 05:45:55.895321 13870 leader_election.cc:302] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c98d6f2af21b4fd19fc7ab8514206bd8; no voters:
I20260926 05:45:55.895470 13870 leader_election.cc:288] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:55.895512 13875 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:55.895627 13875 raft_consensus.cc:686] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [term 1 LEADER]: Becoming Leader. State: Replica: c98d6f2af21b4fd19fc7ab8514206bd8, State: Running, Role: LEADER
I20260926 05:45:55.895757 13870 sys_catalog.cc:564] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:55.895785 13875 consensus_queue.cc:237] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45025 } }
I20260926 05:45:55.896122 13875 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c98d6f2af21b4fd19fc7ab8514206bd8. Latest consensus state: current_term: 1 leader_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45025 } } }
I20260926 05:45:55.896275 13875 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:55.897197 13884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:55.897276 13877 sys_catalog.cc:455] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c98d6f2af21b4fd19fc7ab8514206bd8" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 45025 } } }
I20260926 05:45:55.897356 13877 sys_catalog.cc:458] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:55.897738 13884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:55.899492 13884 catalog_manager.cc:1347] Generated new cluster ID: aaa277cf677c457dbff1b8d820cc29e1
I20260926 05:45:55.899544 13884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 05:45:55 dist-test-slave-1fdt krb5kdc[13789](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401555, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:55.904100 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:55.892328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46753 (local address 127.23.213.254:45025)
0926 05:45:55.892527 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:55.892536 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:55.892569 (+ 33us) server_negotiation.cc:407] Connection header received
0926 05:45:55.893735 (+ 1166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:55.893777 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:55.893837 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:55.893985 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:55.894704 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:55.895875 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:55.896988 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:55.897623 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:55.897735 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:55.900461 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:55.900465 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:55.900466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:55.900477 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:55.902467 (+ 1990us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:55.902847 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:55.902851 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:55.902851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:55.902982 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:55.903181 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:55.903182 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:55.903182 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:55.903510 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:55.903685 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:55.903755 (+ 70us) server_negotiation.cc:299] Negotiation successful
0926 05:45:55.903894 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":92,"threads_started":1}
I20260926 05:45:55.905089 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:55.917205 13874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:55.910054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33475 (local address 127.23.213.254:45025)
0926 05:45:55.910092 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:55.910094 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:55.910244 (+ 150us) server_negotiation.cc:407] Connection header received
0926 05:45:55.910302 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:55.910302 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:55.910313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:55.910370 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:55.911005 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:55.911755 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:55.912767 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:55.913071 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:55.913127 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:55.914526 (+ 1399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:55.914527 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:55.914527 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:55.914535 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:55.916063 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:55.916441 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:55.916442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:55.916443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:55.916493 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:55.916698 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:55.916698 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:55.916699 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:55.916816 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:55.916902 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:55.917036 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 05:45:55.917108 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":12,"server-negotiator.queue_time_us":21}
I20260926 05:45:55.929801 13884 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:55.930354 13884 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:55.936167 13884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c98d6f2af21b4fd19fc7ab8514206bd8: Generated new TSK 0
I20260926 05:45:55.936376 13884 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:55.940829 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13802
2026-09-26T05:45:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1359 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:55 dist-test-slave-1fdt krb5kdc[13901](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:55 dist-test-slave-1fdt krb5kdc[13901](info): set up 2 sockets
Sep 26 05:45:55 dist-test-slave-1fdt krb5kdc[13901](info): commencing operation
krb5kdc: starting...
W20260926 05:45:56.974054 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:57 dist-test-slave-1fdt krb5kdc[13901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401557, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:57.092466 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33597
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:42351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:33597
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:57.172731 13914 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:57.172943 13914 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:57.172966 13914 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:57.174183 13914 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:57.174226 13914 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:57.174240 13914 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:57.174253 13914 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:57.174311 13914 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:57.175772 13914 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:42351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:33597
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:33597
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13914
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:57.176031 13914 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:57.176272 13914 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:57.181077 13922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:57.181082 13919 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:57.181207 13920 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:57.181313 13914 server_base.cc:1034] running on GCE node
I20260926 05:45:57.181587 13914 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:57.181908 13914 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:57.183072 13914 hybrid_clock.cc:648] HybridClock initialized: now 1790401557183049 us; error 53 us; skew 500 ppm
Sep 26 05:45:57 dist-test-slave-1fdt krb5kdc[13901](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401557, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:57.188977 13914 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:57.189471 13914 webserver.cc:460] Webserver started at http://127.23.213.254:35697/ using document root <none> and password file <none>
I20260926 05:45:57.189714 13914 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:57.189805 13914 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:57.189960 13914 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:57.191026 13914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "3e7f02fc86b448b49e67f1e573e906bb"
format_stamp: "Formatted at 2026-09-26 05:45:57 on dist-test-slave-1fdt"
server_key: "aa976e87b75a9ce498be7eef11b08c03"
server_key_iv: "732726f6e46099deba8346605f52baf7"
server_key_version: "encryptionkey@0"
I20260926 05:45:57.191402 13914 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "3e7f02fc86b448b49e67f1e573e906bb"
format_stamp: "Formatted at 2026-09-26 05:45:57 on dist-test-slave-1fdt"
server_key: "aa976e87b75a9ce498be7eef11b08c03"
server_key_iv: "732726f6e46099deba8346605f52baf7"
server_key_version: "encryptionkey@0"
I20260926 05:45:57.192713 13914 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:57.193428 13929 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:57.193595 13914 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:57.193687 13914 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "3e7f02fc86b448b49e67f1e573e906bb"
format_stamp: "Formatted at 2026-09-26 05:45:57 on dist-test-slave-1fdt"
server_key: "aa976e87b75a9ce498be7eef11b08c03"
server_key_iv: "732726f6e46099deba8346605f52baf7"
server_key_version: "encryptionkey@0"
I20260926 05:45:57.193802 13914 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:57.227063 13914 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:57.229696 13914 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:57.229893 13914 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:57.234052 13914 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:33597
I20260926 05:45:57.234100 13981 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:33597 every 8 connection(s)
I20260926 05:45:57.234474 13914 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:57.235356 13982 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:57.237452 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 13914
I20260926 05:45:57.237592 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:57.237820 24407 external_mini_cluster.cc:1428] Setting key 80bd44ad9d70b6ceb29454c53b9aa629
I20260926 05:45:57.237818 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb: Bootstrap starting.
I20260926 05:45:57.238576 13982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:57.238885 13982 log.cc:824] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:57.239704 13982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb: No bootstrap required, opened a new log
I20260926 05:45:57.241997 13982 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7f02fc86b448b49e67f1e573e906bb" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33597 } }
I20260926 05:45:57.242139 13982 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:57.242162 13982 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e7f02fc86b448b49e67f1e573e906bb, State: Initialized, Role: FOLLOWER
I20260926 05:45:57.242309 13982 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7f02fc86b448b49e67f1e573e906bb" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33597 } }
I20260926 05:45:57.242411 13982 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:57.242443 13982 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:57.242547 13982 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:57.243294 13982 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7f02fc86b448b49e67f1e573e906bb" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33597 } }
I20260926 05:45:57.243427 13982 leader_election.cc:302] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3e7f02fc86b448b49e67f1e573e906bb; no voters:
I20260926 05:45:57.243594 13982 leader_election.cc:288] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:57.243636 13987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:57.243808 13987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [term 1 LEADER]: Becoming Leader. State: Replica: 3e7f02fc86b448b49e67f1e573e906bb, State: Running, Role: LEADER
I20260926 05:45:57.243866 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:57.238371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58575 (local address 127.23.213.254:33597)
0926 05:45:57.238548 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:57.238563 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:57.238588 (+ 25us) server_negotiation.cc:407] Connection header received
0926 05:45:57.239166 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:57.239175 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:57.239240 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:57.239392 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:57.239620 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:57.240531 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:57.241722 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:57.242246 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:57.242371 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:57.243067 (+ 696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:57.243070 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:57.243070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:57.243244 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:57.243274 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:57.243286 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:57.243334 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:57.243557 (+ 223us) server_negotiation.cc:299] Negotiation successful
0926 05:45:57.243639 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260926 05:45:57.243947 13987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7f02fc86b448b49e67f1e573e906bb" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33597 } }
I20260926 05:45:57.243841 13982 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:57.244298 13989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3e7f02fc86b448b49e67f1e573e906bb. Latest consensus state: current_term: 1 leader_uuid: "3e7f02fc86b448b49e67f1e573e906bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7f02fc86b448b49e67f1e573e906bb" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33597 } } }
I20260926 05:45:57.244444 13989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:57.244306 13988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3e7f02fc86b448b49e67f1e573e906bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3e7f02fc86b448b49e67f1e573e906bb" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 33597 } } }
I20260926 05:45:57.244648 13988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:57.244935 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:57.244962 13991 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:57.245425 13991 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:57.247287 13991 catalog_manager.cc:1347] Generated new cluster ID: 2212c5b317b54fb4a9040afa090648a1
I20260926 05:45:57.247342 13991 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:57.250303 13985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:57.246492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41961 (local address 127.23.213.254:33597)
0926 05:45:57.246531 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:57.246533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:57.246811 (+ 278us) server_negotiation.cc:407] Connection header received
0926 05:45:57.246859 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:57.246860 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:57.246870 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:57.246970 (+ 100us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:57.247175 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:57.247883 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:57.248868 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:57.249173 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:57.249221 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:57.250010 (+ 789us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:57.250011 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:57.250012 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:57.250016 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:57.250030 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:57.250034 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:57.250078 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:57.250211 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 05:45:57.250230 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:45:57.282634 13991 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:57.283205 13991 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:57.285333 13991 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3e7f02fc86b448b49e67f1e573e906bb: Generated new TSK 0
I20260926 05:45:57.285519 13991 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:57.290710 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 13914
2026-09-26T05:45:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1350 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:57 dist-test-slave-1fdt krb5kdc[14013](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:57 dist-test-slave-1fdt krb5kdc[14013](info): set up 2 sockets
Sep 26 05:45:57 dist-test-slave-1fdt krb5kdc[14013](info): commencing operation
krb5kdc: starting...
W20260926 05:45:58.321913 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.013s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:58 dist-test-slave-1fdt krb5kdc[14013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401558, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:58.439114 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40339
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:40101
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:40339
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:58.522816 14026 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:58.523051 14026 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:58.523075 14026 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:58.524411 14026 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:58.524464 14026 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:58.524479 14026 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:58.524493 14026 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:58.524507 14026 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:58.526114 14026 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:40101
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:40339
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:40339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14026
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:58.526387 14026 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:58.526636 14026 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:58.531724 14031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:58.531703 14032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:58.531745 14034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:58.531885 14026 server_base.cc:1034] running on GCE node
I20260926 05:45:58.532220 14026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:58.532482 14026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:58.533591 14026 hybrid_clock.cc:648] HybridClock initialized: now 1790401558533577 us; error 27 us; skew 500 ppm
Sep 26 05:45:58 dist-test-slave-1fdt krb5kdc[14013](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401558, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:58.536019 14026 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:58.536444 14026 webserver.cc:460] Webserver started at http://127.23.213.254:41115/ using document root <none> and password file <none>
I20260926 05:45:58.536674 14026 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:58.536744 14026 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:58.536898 14026 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:58.538033 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "65f01b7a4bd240b6a3954d63c73acb69"
format_stamp: "Formatted at 2026-09-26 05:45:58 on dist-test-slave-1fdt"
server_key: "ac4a605ba1a262f3512f78a59db3bdd5"
server_key_iv: "c55594bdcb8d30bbf18593c45bb620a0"
server_key_version: "encryptionkey@0"
I20260926 05:45:58.538394 14026 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "65f01b7a4bd240b6a3954d63c73acb69"
format_stamp: "Formatted at 2026-09-26 05:45:58 on dist-test-slave-1fdt"
server_key: "ac4a605ba1a262f3512f78a59db3bdd5"
server_key_iv: "c55594bdcb8d30bbf18593c45bb620a0"
server_key_version: "encryptionkey@0"
I20260926 05:45:58.539702 14026 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:58.540441 14041 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:58.540635 14026 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:45:58.540745 14026 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "65f01b7a4bd240b6a3954d63c73acb69"
format_stamp: "Formatted at 2026-09-26 05:45:58 on dist-test-slave-1fdt"
server_key: "ac4a605ba1a262f3512f78a59db3bdd5"
server_key_iv: "c55594bdcb8d30bbf18593c45bb620a0"
server_key_version: "encryptionkey@0"
I20260926 05:45:58.540817 14026 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:58.544111 14026 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:58.546015 14026 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:58.546166 14026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:58.550261 14026 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:40339
I20260926 05:45:58.550302 14093 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:40339 every 8 connection(s)
I20260926 05:45:58.550638 14026 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:58.551672 14094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:58.553956 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 14026
I20260926 05:45:58.554075 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:58.554224 14094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69: Bootstrap starting.
I20260926 05:45:58.554253 24407 external_mini_cluster.cc:1428] Setting key 86604a718b8848d97b05528fb79997ff
I20260926 05:45:58.554880 14094 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69: Neither blocks nor log segments found. Creating new log.
I20260926 05:45:58.555204 14094 log.cc:824] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69: Log is configured to *not* fsync() on all Append() calls
I20260926 05:45:58.555910 14094 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69: No bootstrap required, opened a new log
I20260926 05:45:58.556823 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:58.554790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33539 (local address 127.23.213.254:40339)
0926 05:45:58.555002 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:58.555021 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:58.555049 (+ 28us) server_negotiation.cc:407] Connection header received
0926 05:45:58.555544 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:58.555552 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:58.555578 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:58.555736 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:58.556410 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:58.556413 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:58.556413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:58.556418 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:58.556468 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:58.556478 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:58.556574 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:58.556608 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 05:45:58.556650 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":88,"threads_started":1}
I20260926 05:45:58.557860 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:45:58.558332 14094 raft_consensus.cc:348] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f01b7a4bd240b6a3954d63c73acb69" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40339 } }
I20260926 05:45:58.558488 14094 raft_consensus.cc:374] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 05:45:58.558532 14094 raft_consensus.cc:729] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65f01b7a4bd240b6a3954d63c73acb69, State: Initialized, Role: FOLLOWER
I20260926 05:45:58.558662 14094 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f01b7a4bd240b6a3954d63c73acb69" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40339 } }
I20260926 05:45:58.558769 14094 raft_consensus.cc:388] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 05:45:58.558828 14094 raft_consensus.cc:482] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 05:45:58.558882 14094 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:45:58.559664 14094 raft_consensus.cc:504] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f01b7a4bd240b6a3954d63c73acb69" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40339 } }
I20260926 05:45:58.559806 14094 leader_election.cc:302] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65f01b7a4bd240b6a3954d63c73acb69; no voters:
I20260926 05:45:58.559968 14094 leader_election.cc:288] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 05:45:58.560111 14104 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:45:58.560225 14094 sys_catalog.cc:564] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:45:58.560281 14104 raft_consensus.cc:686] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [term 1 LEADER]: Becoming Leader. State: Replica: 65f01b7a4bd240b6a3954d63c73acb69, State: Running, Role: LEADER
I20260926 05:45:58.560365 14104 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f01b7a4bd240b6a3954d63c73acb69" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40339 } }
I20260926 05:45:58.560448 14097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:58.559271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45407 (local address 127.23.213.254:40339)
0926 05:45:58.559355 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:58.559357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:58.559432 (+ 75us) server_negotiation.cc:407] Connection header received
0926 05:45:58.559504 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:58.559505 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:58.559509 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:58.559557 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:58.560221 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:58.560222 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:58.560222 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 05:45:58.560226 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:58.560242 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 05:45:58.560245 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:58.560273 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:58.560379 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 05:45:58.560391 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 05:45:58.560817 14104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65f01b7a4bd240b6a3954d63c73acb69. Latest consensus state: current_term: 1 leader_uuid: "65f01b7a4bd240b6a3954d63c73acb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f01b7a4bd240b6a3954d63c73acb69" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40339 } } }
I20260926 05:45:58.560823 14105 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65f01b7a4bd240b6a3954d63c73acb69" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65f01b7a4bd240b6a3954d63c73acb69" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 40339 } } }
I20260926 05:45:58.560920 14104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [sys.catalog]: This master's current role is: LEADER
I20260926 05:45:58.560938 14105 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69 [sys.catalog]: This master's current role is: LEADER
W20260926 05:45:58.562249 14120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:45:58.562312 14120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:45:58.562368 14113 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:45:58.562546 14113 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:45:58.563934 14113 catalog_manager.cc:1347] Generated new cluster ID: 540f290526f9485b8663c3fd333d3cb0
I20260926 05:45:58.563982 14113 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:45:58.574817 14113 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:45:58.574988 14113 catalog_manager.cc:1504] Loading token signing keys...
I20260926 05:45:58.582490 14113 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 65f01b7a4bd240b6a3954d63c73acb69: Generated new TSK 0
I20260926 05:45:58.582695 14113 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:45:58.588846 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 14026
2026-09-26T05:45:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1297 ms)
[----------] 18 tests from AuthTokenIssuingTest (23692 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:45:58 dist-test-slave-1fdt krb5kdc[14125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:45:58 dist-test-slave-1fdt krb5kdc[14125](info): set up 2 sockets
Sep 26 05:45:58 dist-test-slave-1fdt krb5kdc[14125](info): commencing operation
krb5kdc: starting...
W20260926 05:45:59.629974 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:45:59 dist-test-slave-1fdt krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:45:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:45:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:45:59.744911 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:42073
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:42073,127.23.213.253:33913,127.23.213.252:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:45:59.825205 14138 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:45:59.825476 14138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:59.825508 14138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:59.825551 14138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:59.826762 14138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:59.826840 14138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:59.826864 14138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:59.826885 14138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:59.826917 14138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:59.828421 14138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:42073,127.23.213.253:33913,127.23.213.252:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:42073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:59.828711 14138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:59.828944 14138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:59.834007 14144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:59.834013 14143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:59.834028 14146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:59.834331 14138 server_base.cc:1034] running on GCE node
I20260926 05:45:59.834578 14138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:59.834818 14138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:59.835958 14138 hybrid_clock.cc:648] HybridClock initialized: now 1790401559835941 us; error 32 us; skew 500 ppm
Sep 26 05:45:59 dist-test-slave-1fdt krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:59.838449 14138 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:45:59.838831 14138 webserver.cc:460] Webserver started at http://127.23.213.254:44263/ using document root <none> and password file <none>
I20260926 05:45:59.839013 14138 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:59.839053 14138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:59.839151 14138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:59.840251 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "6440f9db1e894f30bd8ad61e7a51f3aa"
format_stamp: "Formatted at 2026-09-26 05:45:59 on dist-test-slave-1fdt"
server_key: "34ea418f6159b408985dfcd6c1a12af6"
server_key_iv: "cf53390fb2cd3b39c78ee366bd76d12b"
server_key_version: "encryptionkey@0"
I20260926 05:45:59.840546 14138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "6440f9db1e894f30bd8ad61e7a51f3aa"
format_stamp: "Formatted at 2026-09-26 05:45:59 on dist-test-slave-1fdt"
server_key: "34ea418f6159b408985dfcd6c1a12af6"
server_key_iv: "cf53390fb2cd3b39c78ee366bd76d12b"
server_key_version: "encryptionkey@0"
I20260926 05:45:59.841856 14138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 05:45:59.842576 14153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:59.842777 14138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 05:45:59.842885 14138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "6440f9db1e894f30bd8ad61e7a51f3aa"
format_stamp: "Formatted at 2026-09-26 05:45:59 on dist-test-slave-1fdt"
server_key: "34ea418f6159b408985dfcd6c1a12af6"
server_key_iv: "cf53390fb2cd3b39c78ee366bd76d12b"
server_key_version: "encryptionkey@0"
I20260926 05:45:59.842965 14138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:59.857050 14138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:45:59.859539 14138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:45:59.859719 14138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:45:59.863644 14138 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:42073
I20260926 05:45:59.863680 14205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:42073 every 8 connection(s)
I20260926 05:45:59.864212 14138 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:45:59.864894 14206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:45:59.866534 14206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:45:59.869962 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 14138
I20260926 05:45:59.870077 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:45:59.870230 24407 external_mini_cluster.cc:1428] Setting key 1ec06ba54b739e22b277d6fceb8b00dc
Sep 26 05:45:59 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:45:59.876542 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.867239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40090 (local address 127.23.213.254:42073)
0926 05:45:59.867414 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 05:45:59.867425 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 05:45:59.867529 (+ 104us) server_negotiation.cc:407] Connection header received
0926 05:45:59.868001 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:45:59.868003 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:45:59.868018 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:45:59.868103 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:45:59.868625 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:59.869435 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:59.870679 (+ 1244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:45:59.870984 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:45:59.871067 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:59.873286 (+ 2219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:45:59.873292 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:45:59.873292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:45:59.873298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:45:59.874939 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:59.875428 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:59.875430 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:59.875430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:59.875506 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:45:59.875780 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:45:59.875780 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:45:59.875781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:45:59.875981 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:45:59.876042 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:45:59.876287 (+ 245us) server_negotiation.cc:299] Negotiation successful
0926 05:45:59.876381 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":112,"threads_started":1}
I20260926 05:45:59.876631 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.867225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:42073 (local address 127.0.0.1:40090)
0926 05:45:59.867402 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:59.867439 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 05:45:59.867881 (+ 442us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:45:59.868107 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:45:59.868110 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:45:59.868383 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:45:59.868539 (+ 156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:59.868543 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:59.869517 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:59.869518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:59.870595 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:45:59.870598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:45:59.871094 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:45:59.871095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:45:59.871183 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:45:59.871673 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:45:59.871685 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:45:59.873191 (+ 1506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:45:59.875025 (+ 1834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:59.875033 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:59.875035 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:59.875342 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:59.875575 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:45:59.875576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:45:59.875577 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:45:59.875708 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:45:59.876049 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:45:59.876057 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:45:59.876146 (+ 89us) client_negotiation.cc:769] Sending connection context
0926 05:45:59.876254 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 05:45:59.876488 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":59,"thread_start_us":95,"threads_started":1}
I20260926 05:45:59.877406 14206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:45:59.877826 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.877666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53024)
0926 05:45:59.877716 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:59.877794 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:45:59.877969 14156 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
I20260926 05:45:59.878377 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.878292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53028)
0926 05:45:59.878321 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:59.878351 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 05:45:59.878477 14206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:33913: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.213.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab.
Entry for principal kudu/127.23.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab.
I20260926 05:45:59.894495 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:33913
--webserver_interface=127.23.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab
--principal=kudu/127.23.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:42073,127.23.213.253:33913,127.23.213.252:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:45:59.927754 14206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } attempt: 1
I20260926 05:45:59.928277 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.928108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53032)
0926 05:45:59.928176 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:59.928235 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 05:45:59.928687 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.928562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53034)
0926 05:45:59.928619 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:59.928666 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 05:45:59.928799 14206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:33913: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
W20260926 05:45:59.976311 14213 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:45:59.976584 14213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:45:59.976627 14213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:45:59.976686 14213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:45:59.977993 14213 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:45:59.978056 14213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:45:59.978080 14213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:45:59.978109 14213 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:45:59.978137 14213 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:45:59.979657 14213 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:42073,127.23.213.253:33913,127.23.213.252:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab
--principal=kudu/127.23.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:33913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14213
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:45:59.979940 14213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:45:59.980185 14213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:45:59.985114 14219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:59.985330 14213 server_base.cc:1034] running on GCE node
W20260926 05:45:59.985093 14221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:45:59.985085 14218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:45:59.985700 14213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:45:59.986068 14213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:45:59.987272 14213 hybrid_clock.cc:648] HybridClock initialized: now 1790401559987235 us; error 51 us; skew 500 ppm
Sep 26 05:45:59 dist-test-slave-1fdt krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:45:59.989702 14213 init.cc:377] Logged in from keytab as kudu/127.23.213.253@KRBTEST.COM (short username kudu)
I20260926 05:45:59.990128 14213 webserver.cc:460] Webserver started at http://127.23.213.253:35635/ using document root <none> and password file <none>
I20260926 05:45:59.990353 14213 fs_manager.cc:359] Metadata directory not provided
I20260926 05:45:59.990422 14213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:45:59.990571 14213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:45:59.991608 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data/instance:
uuid: "657d1b9f24494337bc6c8bc620ebcbe1"
format_stamp: "Formatted at 2026-09-26 05:45:59 on dist-test-slave-1fdt"
server_key: "fea1fe7f925dc48f2aad46cc9bd85ca4"
server_key_iv: "e312f938f48a1e5590a192008280532d"
server_key_version: "encryptionkey@0"
I20260926 05:45:59.991977 14213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/wal/instance:
uuid: "657d1b9f24494337bc6c8bc620ebcbe1"
format_stamp: "Formatted at 2026-09-26 05:45:59 on dist-test-slave-1fdt"
server_key: "fea1fe7f925dc48f2aad46cc9bd85ca4"
server_key_iv: "e312f938f48a1e5590a192008280532d"
server_key_version: "encryptionkey@0"
I20260926 05:45:59.993330 14213 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:45:59.994134 14228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:59.994302 14213 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 05:45:59.994405 14213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/wal
uuid: "657d1b9f24494337bc6c8bc620ebcbe1"
format_stamp: "Formatted at 2026-09-26 05:45:59 on dist-test-slave-1fdt"
server_key: "fea1fe7f925dc48f2aad46cc9bd85ca4"
server_key_iv: "e312f938f48a1e5590a192008280532d"
server_key_version: "encryptionkey@0"
I20260926 05:45:59.994479 14213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:45:59.996961 14206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } attempt: 2
I20260926 05:45:59.997411 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.997265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53048)
0926 05:45:59.997330 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:59.997371 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260926 05:45:59.997807 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:45:59.997684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53060)
0926 05:45:59.997734 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 05:45:59.997782 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 05:45:59.997923 14206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:33913: Network error: Client connection negotiation failed: client connection to 127.23.213.253:33913: connect: Connection refused (error 111)
I20260926 05:46:00.004690 14213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:00.006758 14213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:00.006880 14213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:00.011018 14213 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.253:33913
I20260926 05:46:00.011042 14280 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.253:33913 every 8 connection(s)
I20260926 05:46:00.011430 14213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
I20260926 05:46:00.012300 14281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:00.013922 14281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:00.019155 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 14213
I20260926 05:46:00.019239 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-1/wal/instance
I20260926 05:46:00.019421 24407 external_mini_cluster.cc:1428] Setting key d48bd455b877eea500876ce6b1f2768e
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:46:00.023157 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.014653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40100 (local address 127.23.213.254:42073)
0926 05:46:00.014729 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.014731 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.014936 (+ 205us) server_negotiation.cc:407] Connection header received
0926 05:46:00.015397 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.015398 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.015407 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.015466 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.016053 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.016527 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.017503 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.017709 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.017859 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.020051 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.020053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.020054 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.020062 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.021642 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.022176 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.022177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.022177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.022232 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.022513 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.022514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.022514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.022621 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.022682 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.023014 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.023083 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 05:46:00.023455 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.014645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:42073 (local address 127.0.0.1:40100)
0926 05:46:00.014810 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.014852 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.015313 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.015518 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.015521 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.015761 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.015959 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.015963 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.016606 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.016607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.017401 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.017403 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.017881 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.017882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.017987 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.018465 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.018473 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.019945 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.021728 (+ 1783us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.021735 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.021737 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.022095 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.022299 (+ 204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.022300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.022301 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.022449 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.022699 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.022705 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.022877 (+ 172us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.022997 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.023192 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260926 05:46:00.024067 14281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.213.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.252@KRBTEST.COM" created.
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:00.034514 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.024382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53076 (local address 127.23.213.253:33913)
0926 05:46:00.024527 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.024530 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.024577 (+ 47us) server_negotiation.cc:407] Connection header received
0926 05:46:00.024634 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.024635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.024656 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.024716 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.025022 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.025690 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.026709 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.027011 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.027064 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.031208 (+ 4144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.031217 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.031217 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.031223 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.033134 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.033536 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.033538 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.033538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.033600 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.033824 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.033824 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.033828 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.033977 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.034042 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.034257 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.034369 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":26,"server-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260926 05:46:00.034533 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.024343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53076)
0926 05:46:00.024509 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.024520 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.024574 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.024722 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.024723 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.024887 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.024955 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.024956 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.025792 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.025793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.026630 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.026633 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.027097 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.027098 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.027182 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.027598 (+ 416us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.027603 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.031107 (+ 3504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.033217 (+ 2110us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.033218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.033219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.033453 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.033665 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.033666 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.033666 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.033738 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.034048 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.034049 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.034101 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.034183 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.034309 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150}
I20260926 05:46:00.035542 14281 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:00.035952 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.035800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42444)
0926 05:46:00.035855 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.035917 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 05:46:00.036124 14230 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
I20260926 05:46:00.036578 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.036459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42450)
0926 05:46:00.036512 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.036547 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 05:46:00.036700 14281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35847: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab.
Entry for principal kudu/127.23.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab.
I20260926 05:46:00.046825 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:35847
--webserver_interface=127.23.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab
--principal=kudu/127.23.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:46149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:42073,127.23.213.253:33913,127.23.213.252:35847
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:46:00.085907 14281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } attempt: 1
I20260926 05:46:00.086401 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.086262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42460)
0926 05:46:00.086326 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.086361 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 05:46:00.086733 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.086644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42466)
0926 05:46:00.086688 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.086712 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:46:00.086829 14281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35847: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
I20260926 05:46:00.089169 14206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } attempt: 3
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:00.096038 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.089479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53078)
0926 05:46:00.089525 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.089535 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.089603 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.089791 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.089792 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.089962 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.090039 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.090040 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.090709 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.090710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.091246 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.091248 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.091664 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.091665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.091747 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.092240 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.092244 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.093436 (+ 1192us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.095148 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.095149 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.095149 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.095309 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.095505 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.095506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.095506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.095555 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.095808 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.095808 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.095847 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.095897 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.095972 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 05:46:00.096118 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.089561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53078 (local address 127.23.213.253:33913)
0926 05:46:00.089626 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.089628 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.089659 (+ 31us) server_negotiation.cc:407] Connection header received
0926 05:46:00.089684 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.089686 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.089696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.089785 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.090120 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.090616 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.091340 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.091571 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.091664 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.093522 (+ 1858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.093524 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.093524 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.093532 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.095051 (+ 1519us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.095380 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.095381 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.095381 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.095441 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.095613 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.095614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.095614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.095745 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.095809 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.095941 (+ 132us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.096020 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:46:00.096513 14206 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:00.096872 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.096767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42468)
0926 05:46:00.096816 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.096849 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 05:46:00.097230 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.097134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42472)
0926 05:46:00.097185 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.097207 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 05:46:00.097316 14206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35847: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
I20260926 05:46:00.128472 14206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } attempt: 1
I20260926 05:46:00.128937 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.128806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42480)
0926 05:46:00.128871 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.128911 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 05:46:00.129273 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.129181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42488)
0926 05:46:00.129226 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.129250 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:46:00.129361 14206 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35847: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
W20260926 05:46:00.131176 14288 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:46:00.131393 14288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:00.131440 14288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:00.131485 14288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:00.132653 14288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:00.132694 14288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:00.132709 14288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:00.132761 14288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:00.132778 14288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:00.134301 14288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:46149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:42073,127.23.213.253:33913,127.23.213.252:35847
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab
--principal=kudu/127.23.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:35847
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14288
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:00.134553 14288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:00.134791 14288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:00.139667 14296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:00.139626 14294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:00.139644 14293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:00.139904 14288 server_base.cc:1034] running on GCE node
I20260926 05:46:00.140230 14288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:00.140517 14288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:00.141690 14288 hybrid_clock.cc:648] HybridClock initialized: now 1790401560141687 us; error 52 us; skew 500 ppm
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:00.144021 14288 init.cc:377] Logged in from keytab as kudu/127.23.213.252@KRBTEST.COM (short username kudu)
I20260926 05:46:00.144474 14288 webserver.cc:460] Webserver started at http://127.23.213.252:34517/ using document root <none> and password file <none>
I20260926 05:46:00.144702 14288 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:00.144771 14288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:00.144923 14288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:00.146039 14288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data/instance:
uuid: "d44afb21ef634da8afb7306029bfb879"
format_stamp: "Formatted at 2026-09-26 05:46:00 on dist-test-slave-1fdt"
server_key: "7091e7be75bda1279ea4f06e692a441d"
server_key_iv: "66dbfe9cfe8b0d5ff9b9e24570653a98"
server_key_version: "encryptionkey@0"
I20260926 05:46:00.146407 14288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/wal/instance:
uuid: "d44afb21ef634da8afb7306029bfb879"
format_stamp: "Formatted at 2026-09-26 05:46:00 on dist-test-slave-1fdt"
server_key: "7091e7be75bda1279ea4f06e692a441d"
server_key_iv: "66dbfe9cfe8b0d5ff9b9e24570653a98"
server_key_version: "encryptionkey@0"
I20260926 05:46:00.147684 14288 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:00.148352 14303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:00.148518 14288 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:00.148627 14288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/wal
uuid: "d44afb21ef634da8afb7306029bfb879"
format_stamp: "Formatted at 2026-09-26 05:46:00 on dist-test-slave-1fdt"
server_key: "7091e7be75bda1279ea4f06e692a441d"
server_key_iv: "66dbfe9cfe8b0d5ff9b9e24570653a98"
server_key_version: "encryptionkey@0"
I20260926 05:46:00.148701 14288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:00.154955 14281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } attempt: 2
I20260926 05:46:00.155336 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.155214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42500)
0926 05:46:00.155273 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.155309 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:46:00.155658 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.155561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42506)
0926 05:46:00.155604 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.155627 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:46:00.155781 14281 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:35847: Network error: Client connection negotiation failed: client connection to 127.23.213.252:35847: connect: Connection refused (error 111)
I20260926 05:46:00.156988 14288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:00.159468 14288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:00.159631 14288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:00.164832 14288 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.252:35847
I20260926 05:46:00.165244 14288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
I20260926 05:46:00.165966 14355 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.252:35847 every 8 connection(s)
I20260926 05:46:00.166122 14356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:00.167752 14356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:00.171769 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 14288
I20260926 05:46:00.171877 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790401292175461-24407-0/minicluster-data/master-2/wal/instance
I20260926 05:46:00.172044 24407 external_mini_cluster.cc:1428] Setting key 5abbcd945f978b0db48eda4443006e37
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:46:00.177639 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.168431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40110 (local address 127.23.213.254:42073)
0926 05:46:00.168483 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.168485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.168716 (+ 231us) server_negotiation.cc:407] Connection header received
0926 05:46:00.169433 (+ 717us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.169435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.169451 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.169534 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.170472 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.171088 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.172141 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.172434 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.172536 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.174754 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.174755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.174755 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.174761 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.176181 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.176635 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.176635 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.176636 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.176684 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.176970 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.176971 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.176971 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.177069 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.177116 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.177454 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.177528 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:46:00.177677 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.168423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:42073 (local address 127.0.0.1:40110)
0926 05:46:00.168588 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.168633 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.169272 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.169634 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.169639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.170053 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.170360 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.170366 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.171177 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.171178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.172031 (+ 853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.172034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.172542 (+ 508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.172544 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.172686 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.173228 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.173243 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.174668 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.176241 (+ 1573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.176248 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.176250 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.176555 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.176750 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.176751 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.176751 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.176898 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.177131 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.177138 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.177307 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.177408 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.177520 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:46:00.178324 14356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:00.181493 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.172447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42512 (local address 127.23.213.252:35847)
0926 05:46:00.172734 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.172738 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.172752 (+ 14us) server_negotiation.cc:407] Connection header received
0926 05:46:00.172769 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.172771 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.172785 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.172869 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.173458 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.174179 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.175161 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.175650 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.175723 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.178668 (+ 2945us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.178675 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.178675 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.178681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.180253 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.180567 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.180570 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.180570 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.180635 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.180854 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.180855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.180855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.180993 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.181157 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.181224 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.181299 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":94,"threads_started":1}
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:00.185521 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.178668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.23.213.253:33913)
0926 05:46:00.178710 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.178712 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.178756 (+ 44us) server_negotiation.cc:407] Connection header received
0926 05:46:00.178830 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.178831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.178840 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.178895 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.179202 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.179767 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.180629 (+ 862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.180825 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.180917 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.183074 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.183075 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.183076 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.183082 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.184382 (+ 1300us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.184721 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.184722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.184722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.184784 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.185019 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.185021 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.185021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.185136 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.185215 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.185360 (+ 145us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.185430 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:46:00.185521 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.178600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:33913 (local address 127.0.0.1:53092)
0926 05:46:00.178711 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.178722 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.178771 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.178917 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.178918 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.179064 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.179118 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.179120 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.179844 (+ 724us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.179846 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.180549 (+ 703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.180551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.180916 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.180917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.180989 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.181498 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.181502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.182998 (+ 1496us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.184482 (+ 1484us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.184483 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.184483 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.184654 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.184856 (+ 202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.184857 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.184857 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.184924 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.185207 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.185208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.185249 (+ 41us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.185335 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.185425 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260926 05:46:00.185838 14356 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401560, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:46:00.190866 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.182332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40112 (local address 127.23.213.254:42073)
0926 05:46:00.182381 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.182383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.182392 (+ 9us) server_negotiation.cc:407] Connection header received
0926 05:46:00.182412 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.182413 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.182427 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.182473 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.183054 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.183745 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.184450 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.184673 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.184753 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.187154 (+ 2401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.187155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.187155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.187161 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.188589 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.189406 (+ 817us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.189408 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.189408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.189467 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.189733 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.189734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.189734 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.189915 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.190675 (+ 760us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.190720 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.190803 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 05:46:00.192333 14358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.186099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42518)
0926 05:46:00.186138 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.186147 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.186180 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.186278 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.186279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.186406 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.186444 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.186446 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.187083 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.187084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.187572 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.187574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.188025 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.188026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.188083 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.188449 (+ 366us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.188454 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.189608 (+ 1154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.191368 (+ 1760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.191368 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.191368 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.191519 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.191738 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.191739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.191739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.191790 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.192032 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.192033 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.192068 (+ 35us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.192135 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.192228 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260926 05:46:00.192382 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.186149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42518 (local address 127.23.213.252:35847)
0926 05:46:00.186189 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.186191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.186206 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:46:00.186222 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.186223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.186230 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.186273 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.186512 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.187000 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.187630 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.187947 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.188025 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.189681 (+ 1656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.189682 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.189683 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.189689 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.191250 (+ 1561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.191617 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.191618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.191618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.191667 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.191860 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.191861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.191862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.191967 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.192040 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.192162 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.192273 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":24}
I20260926 05:46:00.194096 14356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: Bootstrap starting.
I20260926 05:46:00.195308 14356 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:00.195624 14356 log.cc:824] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: Log is configured to *not* fsync() on all Append() calls
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:00.196436 14356 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: No bootstrap required, opened a new log
I20260926 05:46:00.198748 14356 raft_consensus.cc:348] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.198913 14356 raft_consensus.cc:729] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44afb21ef634da8afb7306029bfb879, State: Initialized, Role: FOLLOWER
I20260926 05:46:00.199002 14356 consensus_queue.cc:260] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.199295 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.191609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53102 (local address 127.23.213.253:33913)
0926 05:46:00.191648 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.191650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.191658 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:46:00.191711 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.191711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.191720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.191774 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.192215 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.192885 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.193587 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.193893 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.193962 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.196508 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.196509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.196509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.196516 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.198209 (+ 1693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.198513 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.198514 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.198515 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.198573 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.198758 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.198759 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.198760 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.198884 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.199042 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.199125 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.199215 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:46:00.199325 14356 sys_catalog.cc:564] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:00.200801 14362 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.200914 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:46:00.200958 14362 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:46:00.201399 14373 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:46:00.201464 14373 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:46:00.201965 14180 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40112
I20260926 05:46:00.204504 14206 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } attempt: 2
I20260926 05:46:00.202087 14288 master_runner.cc:186] Error getting master registration for 127.23.213.254:42073: OK, instance_id { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" instance_seqno: 1790401559862514 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:46:00.204916 14288 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:46:00.212549 14208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.204734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42528)
0926 05:46:00.204783 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.204794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.204830 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.204994 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.204995 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.205178 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.205239 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.205241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.206182 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.206183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.206977 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.206980 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.207460 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.207461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.207526 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.208025 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.208030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.209568 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.211408 (+ 1840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.211409 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.211410 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.211656 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.211859 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.211860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.211860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.211924 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.212238 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.212238 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.212286 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.212352 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.212443 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 05:46:00.212563 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.204782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42528 (local address 127.23.213.252:35847)
0926 05:46:00.204830 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.204832 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.204840 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:46:00.204899 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.204900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.204911 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.204956 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.205322 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.206095 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.207077 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.207388 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.207447 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.209665 (+ 2218us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.209667 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.209667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.209674 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.211320 (+ 1646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.211731 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.211732 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.211733 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.211790 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.211990 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.211991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.211991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.212161 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.212238 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.212393 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.212473 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:46:00.214392 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa: Bootstrap starting.
I20260926 05:46:00.215118 14206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:00.215426 14206 log.cc:824] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:00.216270 14206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa: No bootstrap required, opened a new log
I20260926 05:46:00.217618 14206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.217784 14206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6440f9db1e894f30bd8ad61e7a51f3aa, State: Initialized, Role: FOLLOWER
I20260926 05:46:00.217875 14206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.218183 14206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:00.218155 14376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.218221 14180 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 05:46:00.218243 14376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:00.218267 14180 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:00.219143 14180 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.219460 14180 leader_election.cc:288] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [CANDIDATE]: Term 1 election: Requested vote from peers 657d1b9f24494337bc6c8bc620ebcbe1 (127.23.213.253:33913), d44afb21ef634da8afb7306029bfb879 (127.23.213.252:35847)
I20260926 05:46:00.220352 14330 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "d44afb21ef634da8afb7306029bfb879"
I20260926 05:46:00.220487 14330 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:00.221199 14330 raft_consensus.cc:2454] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6440f9db1e894f30bd8ad61e7a51f3aa in term 1.
I20260926 05:46:00.221454 14155 leader_election.cc:302] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6440f9db1e894f30bd8ad61e7a51f3aa, d44afb21ef634da8afb7306029bfb879; no voters:
I20260926 05:46:00.221576 14376 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:00.221717 14376 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [term 1 LEADER]: Becoming Leader. State: Replica: 6440f9db1e894f30bd8ad61e7a51f3aa, State: Running, Role: LEADER
I20260926 05:46:00.221843 14376 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.222277 14392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6440f9db1e894f30bd8ad61e7a51f3aa. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.222389 14392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:00.222990 14394 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:00.220693 14138 master_runner.cc:186] Error getting master registration for 127.23.213.253:33913: OK, instance_id { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" instance_seqno: 1790401560009818 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:46:00.223382 14138 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:46:00.223470 14394 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:00.225057 14330 raft_consensus.cc:1261] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [term 1 FOLLOWER]: Refusing update from remote peer 6440f9db1e894f30bd8ad61e7a51f3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:00.225343 14392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:00.226619 14362 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6440f9db1e894f30bd8ad61e7a51f3aa. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.226711 14362 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:00.228188 14392 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.228268 14392 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:00.228732 14362 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.228816 14362 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:00.229035 14396 mvcc.cc:205] Tried to move back new op lower bound from 7333484790678999040 to 7333484790669271040. Current Snapshot: MvccSnapshot[applied={T|T < 7333484790678999040}]
I20260926 05:46:00.229854 14400 mvcc.cc:205] Tried to move back new op lower bound from 7333484790678999040 to 7333484790669271040. Current Snapshot: MvccSnapshot[applied={T|T < 7333484790678999040}]
I20260926 05:46:00.235456 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.224180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42540 (local address 127.23.213.252:35847)
0926 05:46:00.224215 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.224216 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.224791 (+ 575us) server_negotiation.cc:407] Connection header received
0926 05:46:00.224890 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.224891 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.224901 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.224930 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.225703 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.226465 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.227532 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.227835 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.227874 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.229475 (+ 1601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.229476 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.229476 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.229483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.231239 (+ 1756us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.233088 (+ 1849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.233089 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.233090 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.233170 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.234076 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.234077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.234077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.234232 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.234325 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.235245 (+ 920us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.235349 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 05:46:00.235775 14394 catalog_manager.cc:1347] Generated new cluster ID: 80558c6c737f4f07adfcab2e6f2859e3
I20260926 05:46:00.235826 14394 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:46:00.246932 14209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.223960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40124 (local address 127.23.213.254:42073)
0926 05:46:00.224000 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.224002 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.230448 (+ 6446us) server_negotiation.cc:407] Connection header received
0926 05:46:00.231837 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.231838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.231850 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.231910 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.233096 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.233865 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.237596 (+ 3731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.237926 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.237968 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.241228 (+ 3260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.241230 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.241230 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.241237 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.242902 (+ 1665us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.245346 (+ 2444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.245347 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.245348 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.245403 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.245897 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.245898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.245898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.246026 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.246098 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.246749 (+ 651us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.246833 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:46:00.247015 14281 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } attempt: 3
I20260926 05:46:00.247798 14285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.224625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53112 (local address 127.23.213.253:33913)
0926 05:46:00.224679 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.224681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.231724 (+ 7043us) server_negotiation.cc:407] Connection header received
0926 05:46:00.231868 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.231870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.231882 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.231913 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.235497 (+ 3584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.236200 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.238046 (+ 1846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.238356 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.238395 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.243307 (+ 4912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.243308 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.243309 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.243316 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.244931 (+ 1615us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.245607 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.245608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.245609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.245670 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.246341 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.246342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.246342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.246476 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.246549 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.247608 (+ 1059us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.247707 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:46:00.248297 14394 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:00.249704 14394 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 05:46:00 dist-test-slave-1fdt krb5kdc[14125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401559, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:46:00.257098 14283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.247391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:35847 (local address 127.0.0.1:42556)
0926 05:46:00.247469 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:00.247480 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:00.247517 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:00.247960 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:00.247961 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:00.248148 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:00.248195 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.248197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.249319 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.249320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.250064 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:00.250067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.250592 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.250592 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:00.250645 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.251117 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:00.251123 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:00.252615 (+ 1492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:00.255885 (+ 3270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.255885 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.255886 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.256094 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.256373 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:00.256374 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:00.256374 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:00.256440 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:00.256794 (+ 354us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:00.256794 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:00.256847 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 05:46:00.256893 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 05:46:00.256981 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260926 05:46:00.257845 14360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:00.247588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42556 (local address 127.23.213.252:35847)
0926 05:46:00.247824 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.247826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.247834 (+ 8us) server_negotiation.cc:407] Connection header received
0926 05:46:00.247855 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.247856 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.247866 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.247916 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.248502 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.249228 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.250197 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:00.250487 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:00.250542 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:00.252690 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:00.252692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:00.252692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:00.252699 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:00.255770 (+ 3071us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.256185 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.256186 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.256186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.256284 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:00.256515 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:00.256515 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:00.256515 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:00.256705 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:00.257524 (+ 819us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:00.257586 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 05:46:00.257686 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":215}
I20260926 05:46:00.259564 14281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: Bootstrap starting.
I20260926 05:46:00.260275 14281 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:00.260571 14281 log.cc:824] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:00.262274 14281 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: No bootstrap required, opened a new log
I20260926 05:46:00.264293 14281 raft_consensus.cc:348] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.264431 14281 raft_consensus.cc:729] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 657d1b9f24494337bc6c8bc620ebcbe1, State: Initialized, Role: FOLLOWER
I20260926 05:46:00.264513 14281 consensus_queue.cc:260] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } }
I20260926 05:46:00.264825 14281 sys_catalog.cc:564] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:00.264909 14254 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:00.265799 14255 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "657d1b9f24494337bc6c8bc620ebcbe1"
I20260926 05:46:00.265879 14254 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [term 1 FOLLOWER]: Refusing update from remote peer 6440f9db1e894f30bd8ad61e7a51f3aa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:00.266141 14392 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [LEADER]: Connected to new peer: Peer: permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:00.266652 14255 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6440f9db1e894f30bd8ad61e7a51f3aa in term 1.
I20260926 05:46:00.268272 14420 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6440f9db1e894f30bd8ad61e7a51f3aa. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.268347 14394 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa: Generated new TSK 0
I20260926 05:46:00.268515 14394 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:46:00.270191 14376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.270277 14376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6440f9db1e894f30bd8ad61e7a51f3aa [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:00.269627 14418 mvcc.cc:205] Tried to move back new op lower bound from 7333484790768619520 to 7333484790669271040. Current Snapshot: MvccSnapshot[applied={T|T < 7333484790678999040}]
I20260926 05:46:00.271050 14406 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.272758 14406 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:00.272846 14420 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:00.273550 14413 sys_catalog.cc:455] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6440f9db1e894f30bd8ad61e7a51f3aa" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 42073 } } peers { permanent_uuid: "657d1b9f24494337bc6c8bc620ebcbe1" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 33913 } } peers { permanent_uuid: "d44afb21ef634da8afb7306029bfb879" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 35847 } } }
I20260926 05:46:00.273633 14413 sys_catalog.cc:458] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 05:46:00.274907 14421 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:46:00.274991 14421 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260926 05:46:00.288607 14209 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.287068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40128 (local address 127.23.213.254:42073)
0926 05:46:00.287116 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.287118 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.287898 (+ 780us) server_negotiation.cc:407] Connection header received
0926 05:46:00.288013 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.288015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.288027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.288071 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.288507 (+ 436us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40128: BlockingRecv error: recv got EOF from 127.0.0.1:40128 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:46:00.289726 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.287415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53126 (local address 127.23.213.253:33913)
0926 05:46:00.287452 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.287454 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.289047 (+ 1593us) server_negotiation.cc:407] Connection header received
0926 05:46:00.289163 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.289164 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.289174 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.289203 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.289507 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53126: BlockingRecv error: recv got EOF from 127.0.0.1:53126 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:46:00.290531 14360 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.287728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42558 (local address 127.23.213.252:35847)
0926 05:46:00.287768 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.287770 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.290046 (+ 2276us) server_negotiation.cc:407] Connection header received
0926 05:46:00.290143 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.290144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.290153 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.290182 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.290477 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42558: BlockingRecv error: recv got EOF from 127.0.0.1:42558 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:46:00.297896 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.296969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53142 (local address 127.23.213.253:33913)
0926 05:46:00.297011 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.297013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.297237 (+ 224us) server_negotiation.cc:407] Connection header received
0926 05:46:00.297342 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.297343 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.297361 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.297390 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.297841 (+ 451us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53142: BlockingRecv error: recv got EOF from 127.0.0.1:53142 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 05:46:00.301119 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.300319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53158 (local address 127.23.213.253:33913)
0926 05:46:00.300352 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.300354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.300611 (+ 257us) server_negotiation.cc:407] Connection header received
0926 05:46:00.300711 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.300712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.300722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.300752 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.301065 (+ 313us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53158: BlockingRecv error: recv got EOF from 127.0.0.1:53158 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 05:46:00.304975 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.304209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53172 (local address 127.23.213.253:33913)
0926 05:46:00.304254 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.304256 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.304480 (+ 224us) server_negotiation.cc:407] Connection header received
0926 05:46:00.304589 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.304590 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.304599 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.304628 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.304928 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53172: BlockingRecv error: recv got EOF from 127.0.0.1:53172 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 05:46:00.311017 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.310211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53182 (local address 127.23.213.253:33913)
0926 05:46:00.310248 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.310250 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.310499 (+ 249us) server_negotiation.cc:407] Connection header received
0926 05:46:00.310610 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.310612 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.310622 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.310651 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.310967 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53182: BlockingRecv error: recv got EOF from 127.0.0.1:53182 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 05:46:00.321190 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.320322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53194 (local address 127.23.213.253:33913)
0926 05:46:00.320375 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.320378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.320644 (+ 266us) server_negotiation.cc:407] Connection header received
0926 05:46:00.320745 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.320746 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.320757 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.320786 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.321125 (+ 339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53194: BlockingRecv error: recv got EOF from 127.0.0.1:53194 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 05:46:00.340698 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.339795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53200 (local address 127.23.213.253:33913)
0926 05:46:00.339839 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.339841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.340110 (+ 269us) server_negotiation.cc:407] Connection header received
0926 05:46:00.340216 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.340217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.340229 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.340258 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.340635 (+ 377us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53200: BlockingRecv error: recv got EOF from 127.0.0.1:53200 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:46:00.375286 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.374255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53214 (local address 127.23.213.253:33913)
0926 05:46:00.374302 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.374305 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.374655 (+ 350us) server_negotiation.cc:407] Connection header received
0926 05:46:00.374769 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.374770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.374783 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.374816 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.375223 (+ 407us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53214: BlockingRecv error: recv got EOF from 127.0.0.1:53214 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 05:46:00.441515 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.440786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53222 (local address 127.23.213.253:33913)
0926 05:46:00.440891 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.440894 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.440977 (+ 83us) server_negotiation.cc:407] Connection header received
0926 05:46:00.441025 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.441026 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.441036 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.441090 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.441449 (+ 359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53222: BlockingRecv error: recv got EOF from 127.0.0.1:53222 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260926 05:46:00.571732 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.570831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53232 (local address 127.23.213.253:33913)
0926 05:46:00.570889 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.570891 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.571012 (+ 121us) server_negotiation.cc:407] Connection header received
0926 05:46:00.571102 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.571103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.571117 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.571201 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.571669 (+ 468us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53232: BlockingRecv error: recv got EOF from 127.0.0.1:53232 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:46:00.829676 14285 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:00.829070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53238 (local address 127.23.213.253:33913)
0926 05:46:00.829139 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:00.829141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:00.829151 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:46:00.829199 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:00.829199 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:00.829209 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:00.829269 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:00.829586 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53238: BlockingRecv error: recv got EOF from 127.0.0.1:53238 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:46:01.201879 14373 catalog_manager.cc:1259] Loaded cluster ID: 80558c6c737f4f07adfcab2e6f2859e3
I20260926 05:46:01.201953 14373 catalog_manager.cc:1552] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: loading cluster ID for follower catalog manager: success
I20260926 05:46:01.202777 14373 catalog_manager.cc:1574] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: acquiring CA information for follower catalog manager: success
I20260926 05:46:01.203120 14373 catalog_manager.cc:1602] T 00000000000000000000000000000000 P d44afb21ef634da8afb7306029bfb879: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:46:01.275404 14421 catalog_manager.cc:1259] Loaded cluster ID: 80558c6c737f4f07adfcab2e6f2859e3
I20260926 05:46:01.275473 14421 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: loading cluster ID for follower catalog manager: success
I20260926 05:46:01.276341 14421 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: acquiring CA information for follower catalog manager: success
I20260926 05:46:01.276681 14421 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 657d1b9f24494337bc6c8bc620ebcbe1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:46:01.345619 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:01.342870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53244 (local address 127.23.213.253:33913)
0926 05:46:01.342990 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:01.342992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:01.343074 (+ 82us) server_negotiation.cc:407] Connection header received
0926 05:46:01.343174 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:01.343175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:01.343236 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:01.343329 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:01.343495 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.343992 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.344778 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.344994 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.345072 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:01.345262 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:01.345370 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:01.345459 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:01.345498 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 05:46:01.345518 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260926 05:46:01.349689 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:01.347544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42572 (local address 127.23.213.252:35847)
0926 05:46:01.347641 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:01.347643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:01.347658 (+ 15us) server_negotiation.cc:407] Connection header received
0926 05:46:01.347695 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:01.347696 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:01.347730 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:01.347799 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:01.347860 (+ 61us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.348328 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.348988 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.349185 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.349235 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:01.349355 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:01.349450 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:01.349501 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:01.349540 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 05:46:01.349561 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":55,"threads_started":1}
I20260926 05:46:01.353976 14497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:01.351315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42586 (local address 127.23.213.252:35847)
0926 05:46:01.351355 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:01.351357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:01.351368 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:46:01.351472 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:01.351473 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:01.351484 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:01.351534 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:01.351696 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.352370 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.353205 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.353489 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.353525 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:01.353686 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:01.353717 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:01.353790 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:01.353882 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 05:46:01.353905 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:46:01.354559 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:01.351819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53250 (local address 127.23.213.253:33913)
0926 05:46:01.351864 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:01.351866 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:01.351937 (+ 71us) server_negotiation.cc:407] Connection header received
0926 05:46:01.352011 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:01.352012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:01.352023 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:01.352079 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:01.352219 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.352920 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.353632 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.353950 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.354014 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:01.354307 (+ 293us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:01.354341 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:01.354387 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:01.354465 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 05:46:01.354486 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 05:46:01.354841 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:01.351264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40142 (local address 127.23.213.254:42073)
0926 05:46:01.351379 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:01.351381 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:01.351544 (+ 163us) server_negotiation.cc:407] Connection header received
0926 05:46:01.351595 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:01.351596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:01.351670 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:01.351758 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:01.351939 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.352798 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.353808 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:01.354126 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:01.354226 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:01.354419 (+ 193us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:01.354547 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:01.354592 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:01.354730 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 05:46:01.354757 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":62,"threads_started":1}
I20260926 05:46:01.356556 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 14138
I20260926 05:46:01.361943 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 14213
I20260926 05:46:01.366849 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 14288
2026-09-26T05:46:01Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2780 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 05:46:01 dist-test-slave-1fdt krb5kdc[14508](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 05:46:01 dist-test-slave-1fdt krb5kdc[14508](info): set up 2 sockets
Sep 26 05:46:01 dist-test-slave-1fdt krb5kdc[14508](info): commencing operation
krb5kdc: starting...
W20260926 05:46:02.402014 24407 mini_kdc.cc:121] Time spent starting KDC: real 1.015s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/test-user.keytab.
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T05:46:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T05:46:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.23.213.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
Entry for principal kudu/127.23.213.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab.
I20260926 05:46:02.518447 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38593
--webserver_interface=127.23.213.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38593,127.23.213.253:42711,127.23.213.252:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 05:46:02.598877 14521 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:46:02.599130 14521 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:02.599164 14521 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:02.599205 14521 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:02.600348 14521 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:02.600418 14521 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:02.600445 14521 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:02.600476 14521 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:02.600502 14521 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:02.602017 14521 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38593,127.23.213.253:42711,127.23.213.252:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.254.keytab
--principal=kudu/127.23.213.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.254:38593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.23.213.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14521
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:02.602278 14521 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:02.602501 14521 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:02.607228 14527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:02.607259 14529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:02.607298 14526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:02.607434 14521 server_base.cc:1034] running on GCE node
I20260926 05:46:02.607731 14521 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:02.608012 14521 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:02.609176 14521 hybrid_clock.cc:648] HybridClock initialized: now 1790401562609167 us; error 29 us; skew 500 ppm
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:02.616115 14521 init.cc:377] Logged in from keytab as kudu/127.23.213.254@KRBTEST.COM (short username kudu)
I20260926 05:46:02.616706 14521 webserver.cc:460] Webserver started at http://127.23.213.254:37907/ using document root <none> and password file <none>
I20260926 05:46:02.616972 14521 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:02.617057 14521 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:02.617204 14521 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:02.618618 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data/instance:
uuid: "5e83658aba284b928348fa81b0eb0708"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "4eeb6ef7017dd8e752ab998d99a13ff6"
server_key_iv: "f798abc18d7e5bd44972dfb703137e54"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.619043 14521 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/wal/instance:
uuid: "5e83658aba284b928348fa81b0eb0708"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "4eeb6ef7017dd8e752ab998d99a13ff6"
server_key_iv: "f798abc18d7e5bd44972dfb703137e54"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.625494 14521 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.001s
I20260926 05:46:02.626368 14536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:02.626576 14521 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:02.626659 14521 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/wal
uuid: "5e83658aba284b928348fa81b0eb0708"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "4eeb6ef7017dd8e752ab998d99a13ff6"
server_key_iv: "f798abc18d7e5bd44972dfb703137e54"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.626751 14521 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:02.641487 14521 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:02.643368 14521 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:02.643523 14521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:02.647573 14521 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.254:38593
I20260926 05:46:02.647605 14588 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.254:38593 every 8 connection(s)
I20260926 05:46:02.648006 14521 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/data/info.pb
I20260926 05:46:02.648895 14589 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:02.650432 14589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:02.653877 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 14521
I20260926 05:46:02.653983 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-0/wal/instance
I20260926 05:46:02.654153 24407 external_mini_cluster.cc:1428] Setting key 64c144dd2b57f2cd7881b3a7b38b15dc
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:46:02.660248 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.651058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38593 (local address 127.0.0.1:44230)
0926 05:46:02.651234 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.651276 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.651736 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.651997 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.652000 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.652277 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.652437 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.652441 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.653403 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.653404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.654252 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.654254 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.654933 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.654934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.654990 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.655541 (+ 551us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.655549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.656974 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.658768 (+ 1794us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.658773 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.658775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.658965 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.659222 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.659223 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.659223 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.659309 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.659667 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.659672 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.659732 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.659841 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.660077 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":93,"thread_start_us":89,"threads_started":1}
I20260926 05:46:02.660255 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.651112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44230 (local address 127.23.213.254:38593)
0926 05:46:02.651276 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.651288 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.651356 (+ 68us) server_negotiation.cc:407] Connection header received
0926 05:46:02.651859 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.651861 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.651877 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.651963 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.652519 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.653322 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.654341 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.654845 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.654963 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.657083 (+ 2120us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.657090 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.657093 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.657104 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.658674 (+ 1570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.659060 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.659063 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.659064 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.659143 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.659374 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.659375 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.659376 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.659593 (+ 217us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.659688 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.659873 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.660004 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260926 05:46:02.661096 14589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:02.661434 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.661314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33940)
0926 05:46:02.661363 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.661415 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:46:02.661561 14538 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.23.213.253@KRBTEST.COM; defaulting to no policy
I20260926 05:46:02.662012 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.661913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33944)
0926 05:46:02.661961 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.661984 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:46:02.662110 14589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:42711: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab.
Entry for principal kudu/127.23.213.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab.
I20260926 05:46:02.678481 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:42711
--webserver_interface=127.23.213.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab
--principal=kudu/127.23.213.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38593,127.23.213.253:42711,127.23.213.252:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:46:02.711424 14589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } attempt: 1
I20260926 05:46:02.712065 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.711868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33946)
0926 05:46:02.711977 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.712035 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260926 05:46:02.712458 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.712373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33952)
0926 05:46:02.712417 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.712441 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 05:46:02.712567 14589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:42711: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
W20260926 05:46:02.759788 14596 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:46:02.760007 14596 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:02.760025 14596 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:02.760044 14596 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:02.761152 14596 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:02.761184 14596 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:02.761198 14596 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:02.761210 14596 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:02.761224 14596 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:02.762727 14596 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38593,127.23.213.253:42711,127.23.213.252:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.253.keytab
--principal=kudu/127.23.213.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.253:42711
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.23.213.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14596
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:02.762923 14596 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:02.763087 14596 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:02.767889 14602 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:02.767952 14596 server_base.cc:1034] running on GCE node
W20260926 05:46:02.767848 14604 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:02.767915 14601 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:02.768265 14596 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:02.768522 14596 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:02.769675 14596 hybrid_clock.cc:648] HybridClock initialized: now 1790401562769661 us; error 30 us; skew 500 ppm
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:02.772183 14596 init.cc:377] Logged in from keytab as kudu/127.23.213.253@KRBTEST.COM (short username kudu)
I20260926 05:46:02.772671 14596 webserver.cc:460] Webserver started at http://127.23.213.253:40723/ using document root <none> and password file <none>
I20260926 05:46:02.772912 14596 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:02.772958 14596 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:02.773119 14596 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:02.774245 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data/instance:
uuid: "85818e3048114991a895de41f824c8e5"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "133a0ca696f713fa90b353088b7c260d"
server_key_iv: "6dbc46a0520994a85754d514c2f8f49e"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.774597 14596 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/wal/instance:
uuid: "85818e3048114991a895de41f824c8e5"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "133a0ca696f713fa90b353088b7c260d"
server_key_iv: "6dbc46a0520994a85754d514c2f8f49e"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.775873 14596 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 05:46:02.776590 14611 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:02.776758 14596 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 05:46:02.776863 14596 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/wal
uuid: "85818e3048114991a895de41f824c8e5"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "133a0ca696f713fa90b353088b7c260d"
server_key_iv: "6dbc46a0520994a85754d514c2f8f49e"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.776942 14596 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:02.780702 14589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } attempt: 2
I20260926 05:46:02.781076 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.780950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33966)
0926 05:46:02.781007 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.781043 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:46:02.781411 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.781318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33974)
0926 05:46:02.781362 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.781386 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:46:02.781494 14589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.253:42711: Network error: Client connection negotiation failed: client connection to 127.23.213.253:42711: connect: Connection refused (error 111)
I20260926 05:46:02.790903 14596 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:02.793256 14596 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:02.793408 14596 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:02.797611 14596 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.253:42711
I20260926 05:46:02.797642 14663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.253:42711 every 8 connection(s)
I20260926 05:46:02.798066 14596 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/data/info.pb
I20260926 05:46:02.798915 14664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:02.800449 14664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:02.803295 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 14596
I20260926 05:46:02.803380 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-1/wal/instance
I20260926 05:46:02.803578 24407 external_mini_cluster.cc:1428] Setting key 3910268cbcdd39d0ba997922a1560c27
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:46:02.810668 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.801192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38593 (local address 127.0.0.1:44244)
0926 05:46:02.801372 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.801413 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.801961 (+ 548us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.802235 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.802238 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.802511 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.802719 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.802723 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.803538 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.803539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.804410 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.804413 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.804967 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.804969 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.805129 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.805664 (+ 535us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.805674 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.807441 (+ 1767us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.809310 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.809316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.809318 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.809532 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.809799 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.809800 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.809800 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.809903 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.810196 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.810200 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.810315 (+ 115us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.810391 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.810507 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260926 05:46:02.810669 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.801208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44244 (local address 127.23.213.254:38593)
0926 05:46:02.801278 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.801279 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.801513 (+ 234us) server_negotiation.cc:407] Connection header received
0926 05:46:02.802092 (+ 579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.802094 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.802105 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.802152 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.802796 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.803436 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.804511 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.804843 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.804926 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.807534 (+ 2608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.807537 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.807537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.807546 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.809229 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.809615 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.809616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.809616 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.809686 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.809969 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.809970 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.809971 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.810124 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.810204 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.810457 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.810549 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 05:46:02.811287 14664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.23.213.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.23.213.252@KRBTEST.COM" created.
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:02.818722 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.811546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33982)
0926 05:46:02.811597 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.811607 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.811666 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.811945 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.811946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.812053 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.812101 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.812102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.813129 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.813130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.813655 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.813657 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.814163 (+ 506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.814164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.814217 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.814651 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.814655 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.815787 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.817672 (+ 1885us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.817673 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.817674 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.817896 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.818115 (+ 219us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.818115 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.818115 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.818173 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.818493 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.818494 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.818527 (+ 33us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.818575 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.818653 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:46:02.818892 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.811608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.23.213.253:42711)
0926 05:46:02.811783 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.811788 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.811814 (+ 26us) server_negotiation.cc:407] Connection header received
0926 05:46:02.811835 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.811837 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.811851 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.811938 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.812185 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.812998 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.813725 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.814096 (+ 371us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.814154 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.815863 (+ 1709us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.815871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.815872 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.815879 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.817571 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.817971 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.817974 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.817975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.818042 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.818233 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.818234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.818235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.818430 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.818512 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.818607 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.818727 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":140,"thread_start_us":94,"threads_started":1}
I20260926 05:46:02.819490 14664 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:02.819854 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.819741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36042)
0926 05:46:02.819789 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.819837 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 05:46:02.820003 14613 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
I20260926 05:46:02.820430 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.820336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36056)
0926 05:46:02.820383 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.820407 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 05:46:02.820547 14664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:43891: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.23.213.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab.
Entry for principal kudu/127.23.213.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab.
I20260926 05:46:02.832064 24407 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPYhTSt/build/release/bin/kudu
/tmp/dist-test-taskPYhTSt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:43891
--webserver_interface=127.23.213.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab
--principal=kudu/127.23.213.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.23.213.212:39065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.23.213.254:38593,127.23.213.253:42711,127.23.213.252:43891
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 05:46:02.869876 14664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } attempt: 1
I20260926 05:46:02.870453 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.870286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36058)
0926 05:46:02.870377 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.870421 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260926 05:46:02.870869 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.870785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36062)
0926 05:46:02.870826 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.870851 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:46:02.870988 14664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:43891: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
I20260926 05:46:02.872638 14589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } attempt: 3
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:02.879582 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.872919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:33998)
0926 05:46:02.872974 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.872986 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.873036 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.873207 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.873208 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.873370 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.873433 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.873435 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.874175 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.874176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.874717 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.874719 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.875119 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.875120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.875206 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.875683 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.875688 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.876873 (+ 1185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.878532 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.878533 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.878534 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.878720 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.878933 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.878934 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.878934 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.879000 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.879250 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.879251 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.879297 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.879386 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.879484 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 05:46:02.879578 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.872970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33998 (local address 127.23.213.253:42711)
0926 05:46:02.873011 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.873013 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.873024 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:46:02.873122 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.873123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.873143 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.873197 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.873520 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.874083 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.874812 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.875034 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.875118 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.876961 (+ 1843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.876963 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.876963 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.876971 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.878432 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.878806 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.878807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.878808 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.878865 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.879064 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.879065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.879066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.879181 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.879257 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.879412 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.879484 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:46:02.880146 14589 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:02.880501 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.880404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36070)
0926 05:46:02.880453 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.880482 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 05:46:02.880815 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.880729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36072)
0926 05:46:02.880772 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.880795 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 05:46:02.880904 14589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:43891: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
I20260926 05:46:02.912158 14589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } attempt: 1
I20260926 05:46:02.912654 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.912514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36076)
0926 05:46:02.912579 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.912621 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 05:46:02.912984 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.912903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36090)
0926 05:46:02.912939 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.912959 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 05:46:02.913058 14589 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:43891: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
W20260926 05:46:02.916231 14671 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 05:46:02.916455 14671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 05:46:02.916476 14671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 05:46:02.916497 14671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 05:46:02.917692 14671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 05:46:02.917732 14671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 05:46:02.917822 14671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 05:46:02.917857 14671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 05:46:02.917883 14671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 05:46:02.919370 14671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.23.213.212:39065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.23.213.254:38593,127.23.213.253:42711,127.23.213.252:43891
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/krb5kdc/kudu_127.23.213.252.keytab
--principal=kudu/127.23.213.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.23.213.252:43891
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.23.213.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 04:31:17 UTC on ccycloud.quasar-ybjuvq.root.comops.site
build id 4673669
I20260926 05:46:02.919628 14671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 05:46:02.919880 14671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 05:46:02.924750 14679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:02.924817 14676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 05:46:02.924821 14677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 05:46:02.924989 14671 server_base.cc:1034] running on GCE node
I20260926 05:46:02.925218 14671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 05:46:02.925455 14671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 05:46:02.926594 14671 hybrid_clock.cc:648] HybridClock initialized: now 1790401562926585 us; error 33 us; skew 500 ppm
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 05:46:02.928942 14671 init.cc:377] Logged in from keytab as kudu/127.23.213.252@KRBTEST.COM (short username kudu)
I20260926 05:46:02.929423 14671 webserver.cc:460] Webserver started at http://127.23.213.252:38959/ using document root <none> and password file <none>
I20260926 05:46:02.929665 14671 fs_manager.cc:359] Metadata directory not provided
I20260926 05:46:02.929747 14671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 05:46:02.929909 14671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 05:46:02.931017 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data/instance:
uuid: "7e3b2b238ad2403dbed362a907235a64"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "f138d75e31d20a7c235a3ce3233d8b5d"
server_key_iv: "dfb578363c9cb81f466a5e3771e6a322"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.931380 14671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/wal/instance:
uuid: "7e3b2b238ad2403dbed362a907235a64"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "f138d75e31d20a7c235a3ce3233d8b5d"
server_key_iv: "dfb578363c9cb81f466a5e3771e6a322"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.932678 14671 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 05:46:02.933342 14686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 05:46:02.933502 14671 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 05:46:02.933605 14671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data,/tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/wal
uuid: "7e3b2b238ad2403dbed362a907235a64"
format_stamp: "Formatted at 2026-09-26 05:46:02 on dist-test-slave-1fdt"
server_key: "f138d75e31d20a7c235a3ce3233d8b5d"
server_key_iv: "dfb578363c9cb81f466a5e3771e6a322"
server_key_version: "encryptionkey@0"
I20260926 05:46:02.933681 14671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 05:46:02.939131 14664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } attempt: 2
I20260926 05:46:02.939536 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.939414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36094)
0926 05:46:02.939471 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.939508 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 05:46:02.939877 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.939780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36104)
0926 05:46:02.939825 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.939852 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 05:46:02.939989 14664 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.23.213.252:43891: Network error: Client connection negotiation failed: client connection to 127.23.213.252:43891: connect: Connection refused (error 111)
I20260926 05:46:02.952816 14671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 05:46:02.954698 14671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 05:46:02.954866 14671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 05:46:02.958910 14671 rpc_server.cc:307] RPC server started. Bound to: 127.23.213.252:43891
I20260926 05:46:02.958946 14738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.23.213.252:43891 every 8 connection(s)
I20260926 05:46:02.959299 14671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/data/info.pb
I20260926 05:46:02.960197 14739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 05:46:02.961730 14739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } has no permanent_uuid. Determining permanent_uuid...
I20260926 05:46:02.966985 24407 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPYhTSt/build/release/bin/kudu as pid 14671
I20260926 05:46:02.967109 24407 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPYhTSt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790401292175461-24407-0/minicluster-data/master-2/wal/instance
I20260926 05:46:02.967307 24407 external_mini_cluster.cc:1428] Setting key db12fd741bf82056097016c90917a177
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
I20260926 05:46:02.971086 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.962478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44254 (local address 127.23.213.254:38593)
0926 05:46:02.962560 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.962562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.962776 (+ 214us) server_negotiation.cc:407] Connection header received
0926 05:46:02.963216 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.963217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.963227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.963287 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.963873 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.964455 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.965371 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.965543 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.965604 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.968043 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.968046 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.968047 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.968059 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.969616 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.970070 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.970071 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.970072 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.970134 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.970453 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.970453 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.970454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.970555 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.970603 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.970896 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.970958 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 05:46:02.971328 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.962464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.254:38593 (local address 127.0.0.1:44254)
0926 05:46:02.962616 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.962683 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.963132 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.963365 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.963368 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.963601 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.963798 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.963801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.964533 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.964534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.965288 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.965290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.965650 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.965651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.965790 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.966245 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.966253 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.967943 (+ 1690us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.969705 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.969711 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.969713 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.969985 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.970257 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.970257 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.970258 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.970386 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.970616 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.970622 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.970768 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.970877 (+ 109us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.971057 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260926 05:46:02.971884 14739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:02.977205 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.967703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36106 (local address 127.23.213.252:43891)
0926 05:46:02.967858 (+ 155us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.967863 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.967965 (+ 102us) server_negotiation.cc:407] Connection header received
0926 05:46:02.967986 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.967989 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.968004 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.968086 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.968917 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.969984 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.971043 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.971485 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.971558 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.974288 (+ 2730us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.974295 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.974295 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.974302 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.975798 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.976200 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.976204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.976204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.976286 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.976482 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.976482 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.976483 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.976664 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.976836 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.976882 (+ 46us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.977061 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":80,"threads_started":1}
I20260926 05:46:02.978893 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.972249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34010 (local address 127.23.213.253:42711)
0926 05:46:02.972302 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.972304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.972348 (+ 44us) server_negotiation.cc:407] Connection header received
0926 05:46:02.972438 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.972439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.972446 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.972487 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.972779 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.973219 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.974181 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.974468 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.974554 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.976285 (+ 1731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.976286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.976286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.976293 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.977704 (+ 1411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.978122 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.978123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.978123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.978195 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.978393 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.978394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.978394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.978504 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.978554 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.978723 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.978776 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 05:46:02.978912 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.972169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.253:42711 (local address 127.0.0.1:34010)
0926 05:46:02.972298 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.972309 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.972362 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.972493 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.972494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.972650 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.972708 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.972709 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.973277 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.973278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.974104 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.974107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.974573 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.974574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.974647 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.975024 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.975031 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.976209 (+ 1178us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.977850 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.977851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.977851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.978061 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.978257 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.978258 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.978258 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.978329 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.978579 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.978580 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.978629 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.978708 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.978792 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113}
I20260926 05:46:02.979295 14739 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.254@KRBTEST.COM
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.252@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:46:02.985996 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.978128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44270 (local address 127.23.213.254:38593)
0926 05:46:02.978176 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.978178 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.978188 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:46:02.978210 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.978211 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.978222 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.978274 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.978874 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.979712 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.980531 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.980797 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.980845 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.983215 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.983216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.983217 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.983224 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.984830 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.985173 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.985175 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.985176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.985239 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.985438 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.985439 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.985439 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.985558 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.985738 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.985801 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.985873 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 05:46:02.986049 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.979556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36108)
0926 05:46:02.979640 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.979649 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.979693 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.979803 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.979804 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.979939 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.979969 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.979970 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.980670 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.980671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.981334 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.981337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.981666 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.981667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.981734 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.982066 (+ 332us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.982072 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.983485 (+ 1413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.985064 (+ 1579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.985065 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.985065 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.985223 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.985426 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.985427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.985427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.985478 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.985726 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.985726 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.985779 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.985849 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.985939 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73}
I20260926 05:46:02.986099 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.979609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.23.213.252:43891)
0926 05:46:02.979642 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.979644 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.979685 (+ 41us) server_negotiation.cc:407] Connection header received
0926 05:46:02.979743 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.979744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.979754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.979801 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.980036 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.980601 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.981412 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.981593 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.981647 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.983585 (+ 1938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.983586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.983587 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.983593 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.984984 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.985304 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.985305 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.985305 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.985352 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.985551 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.985551 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.985552 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.985659 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.985733 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.985871 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.985986 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":15}
I20260926 05:46:02.987526 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: Bootstrap starting.
I20260926 05:46:02.988193 14589 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } attempt: 2
I20260926 05:46:02.988461 14739 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:02.988798 14739 log.cc:824] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:02.989655 14739 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: No bootstrap required, opened a new log
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.23.213.253@KRBTEST.COM
I20260926 05:46:02.992019 14739 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
Sep 26 05:46:02 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.254@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:46:02.992183 14739 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3b2b238ad2403dbed362a907235a64, State: Initialized, Role: FOLLOWER
I20260926 05:46:02.992277 14739 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
I20260926 05:46:02.992614 14739 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:02.992602 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:02.993625 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:02.995014 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.986751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34022 (local address 127.23.213.253:42711)
0926 05:46:02.986787 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.986789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.986795 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:46:02.986814 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.986815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.986830 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.986895 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.987358 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.987932 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.988760 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.989045 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.989109 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.991697 (+ 2588us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.991698 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.991699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.991706 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.993403 (+ 1697us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.993846 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.993848 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.993849 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.993905 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.994123 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.994124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.994124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.994267 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.994428 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.994842 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.994914 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 05:46:02.995064 14756 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:46:02.995105 14756 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:46:02.995872 14591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.988410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36124)
0926 05:46:02.988455 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:02.988465 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:02.988514 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:02.988671 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:02.988673 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:02.988829 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:02.988886 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.988888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.989808 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.989809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.990297 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:02.990298 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.990681 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.990682 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:02.990740 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.991148 (+ 408us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:02.991153 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:02.992522 (+ 1369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:02.994291 (+ 1769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.994292 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.994293 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.994528 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.994733 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:02.994734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:02.994734 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:02.994802 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:02.995577 (+ 775us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:02.995578 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:02.995621 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 05:46:02.995689 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 05:46:02.995780 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 05:46:02.996322 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:02.988446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36124 (local address 127.23.213.252:43891)
0926 05:46:02.988479 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:02.988480 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:02.988490 (+ 10us) server_negotiation.cc:407] Connection header received
0926 05:46:02.988583 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:02.988584 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:02.988595 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:02.988662 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:02.988966 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.989634 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.990394 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:02.990594 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:02.990679 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:02.992611 (+ 1932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:02.992613 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:02.992613 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:02.992619 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:02.994204 (+ 1585us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.994607 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.994608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.994609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.994674 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:02.994868 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:02.994869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:02.994870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:02.995006 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:02.995063 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:02.996158 (+ 1095us) server_negotiation.cc:299] Negotiation successful
0926 05:46:02.996235 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 05:46:02.996958 24407 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 05:46:02.998431 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708: Bootstrap starting.
I20260926 05:46:02.995466 14671 master_runner.cc:186] Error getting master registration for 127.23.213.254:38593: OK, instance_id { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" instance_seqno: 1790401562646418 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:46:02.998591 14671 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:46:02.997886 14563 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5e83658aba284b928348fa81b0eb0708"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44270
I20260926 05:46:02.999181 14589 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:02.999454 14589 log.cc:824] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:03.000322 14589 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708: No bootstrap required, opened a new log
I20260926 05:46:03.001721 14589 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
I20260926 05:46:03.001920 14589 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e83658aba284b928348fa81b0eb0708, State: Initialized, Role: FOLLOWER
I20260926 05:46:03.002007 14589 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
I20260926 05:46:03.002323 14589 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:03.002398 14563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 05:46:03.002456 14563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:03.003209 14563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
I20260926 05:46:03.003437 14563 leader_election.cc:288] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [CANDIDATE]: Term 1 election: Requested vote from peers 85818e3048114991a895de41f824c8e5 (127.23.213.253:42711), 7e3b2b238ad2403dbed362a907235a64 (127.23.213.252:43891)
I20260926 05:46:03.004617 14713 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e83658aba284b928348fa81b0eb0708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7e3b2b238ad2403dbed362a907235a64"
I20260926 05:46:03.004743 14713 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:03.005659 14713 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e83658aba284b928348fa81b0eb0708 in term 1.
I20260926 05:46:03.006156 14538 leader_election.cc:302] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e83658aba284b928348fa81b0eb0708, 7e3b2b238ad2403dbed362a907235a64; no voters:
I20260926 05:46:03.002288 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.006335 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:03.006353 14776 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 05:46:03.006433 14776 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [term 1 LEADER]: Becoming Leader. State: Replica: 5e83658aba284b928348fa81b0eb0708, State: Running, Role: LEADER
I20260926 05:46:03.006505 14776 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
I20260926 05:46:03.006124 14521 master_runner.cc:186] Error getting master registration for 127.23.213.253:42711: OK, instance_id { permanent_uuid: "85818e3048114991a895de41f824c8e5" instance_seqno: 1790401562796431 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 05:46:03.009632 14521 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 05:46:03.010476 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e83658aba284b928348fa81b0eb0708. Latest consensus state: current_term: 1 leader_uuid: "5e83658aba284b928348fa81b0eb0708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.010583 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: This master's current role is: LEADER
W20260926 05:46:03.010605 14770 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:46:03.010648 14770 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:46:03.011170 14779 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 05:46:03.011354 14779 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 05:46:03.012890 14713 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [term 1 FOLLOWER]: Refusing update from remote peer 5e83658aba284b928348fa81b0eb0708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:03.013203 14776 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 05:46:03.013535 14592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:03.004709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44278 (local address 127.23.213.254:38593)
0926 05:46:03.004747 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.004748 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.005149 (+ 401us) server_negotiation.cc:407] Connection header received
0926 05:46:03.005235 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.005236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.005259 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.005316 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.006124 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.006761 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.007863 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.008199 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.008265 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:03.010092 (+ 1827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:03.010093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:03.010094 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:03.010101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:03.011943 (+ 1842us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.012349 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.012350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.012350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.012412 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.012646 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.012647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.012647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.012786 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:03.012881 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:03.013343 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 05:46:03.013433 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:46:03.014523 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e83658aba284b928348fa81b0eb0708. Latest consensus state: current_term: 1 leader_uuid: "5e83658aba284b928348fa81b0eb0708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.014614 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:03.015604 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e83658aba284b928348fa81b0eb0708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.015692 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: This master's current role is: LEADER
I20260926 05:46:03.016141 14745 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e83658aba284b928348fa81b0eb0708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.016234 14745 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:03.016374 14782 mvcc.cc:205] Tried to move back new op lower bound from 7333484802098241536 to 7333484802075496448. Current Snapshot: MvccSnapshot[applied={T|T < 7333484802098241536 or (T in {7333484802098241536})}]
I20260926 05:46:03.016928 14779 catalog_manager.cc:1347] Generated new cluster ID: a590e73e6f8944daa99868e8ac452158
I20260926 05:46:03.016974 14779 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 05:46:03.022313 14668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:03.013075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34030 (local address 127.23.213.253:42711)
0926 05:46:03.013118 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.013121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.013283 (+ 162us) server_negotiation.cc:407] Connection header received
0926 05:46:03.014097 (+ 814us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.014098 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.014108 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.014139 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.014733 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.015421 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.016406 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.017158 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.017222 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:03.019305 (+ 2083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:03.019306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:03.019306 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:03.019325 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:03.020786 (+ 1461us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.021318 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.021319 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.021319 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.021378 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.021650 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.021650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.021650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.021791 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:03.021864 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:03.022148 (+ 284us) server_negotiation.cc:299] Negotiation successful
0926 05:46:03.022224 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 05:46:03.024295 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:03.013515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36134 (local address 127.23.213.252:43891)
0926 05:46:03.013550 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.013551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.013701 (+ 150us) server_negotiation.cc:407] Connection header received
0926 05:46:03.013888 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.013889 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.013919 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.013950 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.014648 (+ 698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.015389 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.018722 (+ 3333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.018995 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.019071 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:03.020878 (+ 1807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:03.020880 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:03.020880 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:03.020887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:03.022628 (+ 1741us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.023454 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.023455 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.023456 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.023513 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.023716 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.023717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.023717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.023837 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:03.023912 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:03.024131 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 05:46:03.024211 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 05:46:03.031122 14664 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } attempt: 3
I20260926 05:46:03.034372 14779 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 05:46:03.035348 14779 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 05:46:03 dist-test-slave-1fdt krb5kdc[14508](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790401562, etypes {rep=17 tkt=17 ses=17}, kudu/127.23.213.253@KRBTEST.COM for kudu/127.23.213.252@KRBTEST.COM
I20260926 05:46:03.039691 14666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:03.031357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.23.213.252:43891 (local address 127.0.0.1:36146)
0926 05:46:03.031437 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0926 05:46:03.031448 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 05:46:03.031485 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 05:46:03.031710 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 05:46:03.031711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 05:46:03.031875 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 05:46:03.031918 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:03.031920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.032818 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.032819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:03.033640 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 05:46:03.033642 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.034190 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.034191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 05:46:03.034246 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:03.034720 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 05:46:03.034725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 05:46:03.036252 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 05:46:03.038580 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:03.038581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:03.038582 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:03.038792 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:03.039002 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 05:46:03.039003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 05:46:03.039004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 05:46:03.039083 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 05:46:03.039392 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 05:46:03.039393 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 05:46:03.039448 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 05:46:03.039504 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 05:46:03.039595 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260926 05:46:03.039999 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:03.031553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36146 (local address 127.23.213.252:43891)
0926 05:46:03.031589 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.031591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.031597 (+ 6us) server_negotiation.cc:407] Connection header received
0926 05:46:03.031615 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.031616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.031626 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.031670 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.031999 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.032722 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.033733 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:03.034076 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:03.034135 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:03.036322 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 05:46:03.036323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 05:46:03.036323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 05:46:03.036330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 05:46:03.038507 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.038863 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.038864 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.038865 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.038924 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 05:46:03.039159 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 05:46:03.039160 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 05:46:03.039160 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 05:46:03.039293 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 05:46:03.039351 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:03.039845 (+ 494us) server_negotiation.cc:299] Negotiation successful
0926 05:46:03.039914 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 05:46:03.041455 14664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: Bootstrap starting.
I20260926 05:46:03.042167 14664 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: Neither blocks nor log segments found. Creating new log.
I20260926 05:46:03.042467 14664 log.cc:824] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: Log is configured to *not* fsync() on all Append() calls
I20260926 05:46:03.043840 14664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: No bootstrap required, opened a new log
I20260926 05:46:03.045871 14664 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
I20260926 05:46:03.046010 14664 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85818e3048114991a895de41f824c8e5, State: Initialized, Role: FOLLOWER
I20260926 05:46:03.046094 14664 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } }
I20260926 05:46:03.046442 14664 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 05:46:03.046532 14638 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5e83658aba284b928348fa81b0eb0708" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "85818e3048114991a895de41f824c8e5"
I20260926 05:46:03.046659 14638 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 05:46:03.047591 14638 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5e83658aba284b928348fa81b0eb0708 in term 1.
I20260926 05:46:03.046381 14787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.048066 14787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:03.048621 14779 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708: Generated new TSK 0
I20260926 05:46:03.048866 14779 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 05:46:03.049121 14637 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [term 1 FOLLOWER]: Refusing update from remote peer 5e83658aba284b928348fa81b0eb0708: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 05:46:03.049418 14759 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [LEADER]: Connected to new peer: Peer: permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 05:46:03.050279 14798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 05:46:03.050328 14798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 05:46:03.051689 14787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5e83658aba284b928348fa81b0eb0708. Latest consensus state: current_term: 1 leader_uuid: "5e83658aba284b928348fa81b0eb0708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.051877 14787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:03.051965 14787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5e83658aba284b928348fa81b0eb0708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.052076 14787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 05:46:03.052692 14800 mvcc.cc:205] Tried to move back new op lower bound from 7333484802216984576 to 7333484802075496448. Current Snapshot: MvccSnapshot[applied={T|T < 7333484802098241536}]
W20260926 05:46:03.076958 14592 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.075266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44284 (local address 127.23.213.254:38593)
0926 05:46:03.075316 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.075318 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.076251 (+ 933us) server_negotiation.cc:407] Connection header received
0926 05:46:03.076363 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.076364 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.076376 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.076407 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.076871 (+ 464us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44284: BlockingRecv error: recv got EOF from 127.0.0.1:44284 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 05:46:03.077916 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.075662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.23.213.253:42711)
0926 05:46:03.075702 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.075704 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.077361 (+ 1657us) server_negotiation.cc:407] Connection header received
0926 05:46:03.077457 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.077459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.077469 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.077498 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.077855 (+ 357us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34034: BlockingRecv error: recv got EOF from 127.0.0.1:34034 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:46:03.078784 14743 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.075982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.23.213.252:43891)
0926 05:46:03.076020 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.076022 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.078240 (+ 2218us) server_negotiation.cc:407] Connection header received
0926 05:46:03.078338 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.078339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.078351 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.078380 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.078727 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:36160: BlockingRecv error: recv got EOF from 127.0.0.1:36160 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 05:46:03.081224 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.080447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34046 (local address 127.23.213.253:42711)
0926 05:46:03.080482 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.080484 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.080730 (+ 246us) server_negotiation.cc:407] Connection header received
0926 05:46:03.080834 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.080836 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.080853 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.080881 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.081179 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34046: BlockingRecv error: recv got EOF from 127.0.0.1:34046 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 05:46:03.084327 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.083584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.23.213.253:42711)
0926 05:46:03.083618 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.083619 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.083852 (+ 233us) server_negotiation.cc:407] Connection header received
0926 05:46:03.083947 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.083948 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.083958 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.083989 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.084283 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34058: BlockingRecv error: recv got EOF from 127.0.0.1:34058 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 05:46:03.088364 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.087586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34074 (local address 127.23.213.253:42711)
0926 05:46:03.087619 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.087620 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.087846 (+ 226us) server_negotiation.cc:407] Connection header received
0926 05:46:03.087978 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.087979 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.087989 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.088018 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.088318 (+ 300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34074: BlockingRecv error: recv got EOF from 127.0.0.1:34074 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 05:46:03.094527 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.093658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.23.213.253:42711)
0926 05:46:03.093695 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.093697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.094013 (+ 316us) server_negotiation.cc:407] Connection header received
0926 05:46:03.094109 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.094110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.094120 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.094151 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.094477 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34090: BlockingRecv error: recv got EOF from 127.0.0.1:34090 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 05:46:03.104920 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.104065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34096 (local address 127.23.213.253:42711)
0926 05:46:03.104118 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.104120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.104362 (+ 242us) server_negotiation.cc:407] Connection header received
0926 05:46:03.104462 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.104463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.104474 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.104503 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.104854 (+ 351us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34096: BlockingRecv error: recv got EOF from 127.0.0.1:34096 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:46:03.123283 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.122382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34102 (local address 127.23.213.253:42711)
0926 05:46:03.122425 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.122427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.122717 (+ 290us) server_negotiation.cc:407] Connection header received
0926 05:46:03.122826 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.122827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.122838 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.122870 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.123223 (+ 353us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34102: BlockingRecv error: recv got EOF from 127.0.0.1:34102 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 05:46:03.157881 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.156871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.23.213.253:42711)
0926 05:46:03.156916 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.156918 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.157193 (+ 275us) server_negotiation.cc:407] Connection header received
0926 05:46:03.157313 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.157314 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.157328 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.157360 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.157818 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34114: BlockingRecv error: recv got EOF from 127.0.0.1:34114 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 05:46:03.224025 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.223442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34116 (local address 127.23.213.253:42711)
0926 05:46:03.223493 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.223495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.223543 (+ 48us) server_negotiation.cc:407] Connection header received
0926 05:46:03.223616 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.223617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.223627 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.223668 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.223977 (+ 309us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34116: BlockingRecv error: recv got EOF from 127.0.0.1:34116 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 05:46:03.353928 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.353258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34126 (local address 127.23.213.253:42711)
0926 05:46:03.353335 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.353337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.353434 (+ 97us) server_negotiation.cc:407] Connection header received
0926 05:46:03.353478 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.353479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.353489 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.353561 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.353879 (+ 318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34126: BlockingRecv error: recv got EOF from 127.0.0.1:34126 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 05:46:03.436484 14759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5e83658aba284b928348fa81b0eb0708" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5e83658aba284b928348fa81b0eb0708" member_type: VOTER last_known_addr { host: "127.23.213.254" port: 38593 } } peers { permanent_uuid: "85818e3048114991a895de41f824c8e5" member_type: VOTER last_known_addr { host: "127.23.213.253" port: 42711 } } peers { permanent_uuid: "7e3b2b238ad2403dbed362a907235a64" member_type: VOTER last_known_addr { host: "127.23.213.252" port: 43891 } } }
I20260926 05:46:03.436663 14759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5e83658aba284b928348fa81b0eb0708 [sys.catalog]: This master's current role is: LEADER
W20260926 05:46:03.611842 14668 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 05:46:03.611313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.23.213.253:42711)
0926 05:46:03.611384 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:03.611386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:03.611399 (+ 13us) server_negotiation.cc:407] Connection header received
0926 05:46:03.611429 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:03.611429 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:03.611440 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:03.611505 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 05:46:03.611796 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34134: BlockingRecv error: recv got EOF from 127.0.0.1:34134 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 05:46:03.995630 14756 catalog_manager.cc:1259] Loaded cluster ID: a590e73e6f8944daa99868e8ac452158
I20260926 05:46:03.995719 14756 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: loading cluster ID for follower catalog manager: success
I20260926 05:46:03.996599 14756 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: acquiring CA information for follower catalog manager: success
I20260926 05:46:03.996973 14756 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e3b2b238ad2403dbed362a907235a64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:46:04.050734 14798 catalog_manager.cc:1259] Loaded cluster ID: a590e73e6f8944daa99868e8ac452158
I20260926 05:46:04.050808 14798 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: loading cluster ID for follower catalog manager: success
I20260926 05:46:04.051734 14798 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: acquiring CA information for follower catalog manager: success
I20260926 05:46:04.052062 14798 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 85818e3048114991a895de41f824c8e5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 05:46:04.127842 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:04.125055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34148 (local address 127.23.213.253:42711)
0926 05:46:04.125171 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:04.125173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:04.125239 (+ 66us) server_negotiation.cc:407] Connection header received
0926 05:46:04.125291 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:04.125292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:04.125358 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:04.125448 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:04.125608 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.126194 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.126978 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.127197 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.127290 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:04.127484 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:04.127596 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:04.127683 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:04.127718 (+ 35us) server_negotiation.cc:299] Negotiation successful
0926 05:46:04.127742 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":58,"threads_started":1}
I20260926 05:46:04.132027 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:04.129722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36170 (local address 127.23.213.252:43891)
0926 05:46:04.129869 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:04.129871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:04.129882 (+ 11us) server_negotiation.cc:407] Connection header received
0926 05:46:04.129920 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:04.129921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:04.129957 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:04.130041 (+ 84us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:04.130120 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.130639 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.131318 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.131523 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.131574 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:04.131705 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:04.131808 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:04.131866 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:04.131897 (+ 31us) server_negotiation.cc:299] Negotiation successful
0926 05:46:04.131914 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
I20260926 05:46:04.136711 14877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:04.133888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36182 (local address 127.23.213.252:43891)
0926 05:46:04.133961 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:04.133963 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:04.133970 (+ 7us) server_negotiation.cc:407] Connection header received
0926 05:46:04.134045 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:04.134046 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:04.134056 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:04.134106 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:04.134251 (+ 145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.134762 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.135539 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.135832 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.135903 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:04.136072 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:04.136103 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:04.136161 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:04.136618 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 05:46:04.136643 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 05:46:04.136781 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:04.133665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34162 (local address 127.23.213.253:42711)
0926 05:46:04.133717 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:04.133719 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:04.133784 (+ 65us) server_negotiation.cc:407] Connection header received
0926 05:46:04.133947 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:04.133949 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:04.133962 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:04.134019 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:04.134270 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.134988 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.136023 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.136269 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.136316 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:04.136462 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:04.136511 (+ 49us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:04.136558 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:04.136656 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 05:46:04.136689 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 05:46:04.137108 14884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 05:46:04.133636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44290 (local address 127.23.213.254:38593)
0926 05:46:04.133787 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 05:46:04.133789 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 05:46:04.133942 (+ 153us) server_negotiation.cc:407] Connection header received
0926 05:46:04.134126 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 05:46:04.134127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 05:46:04.134180 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 05:46:04.134259 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 05:46:04.134427 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.135206 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.136140 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 05:46:04.136436 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 05:46:04.136497 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 05:46:04.136723 (+ 226us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 05:46:04.136836 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 05:46:04.136883 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 05:46:04.137000 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 05:46:04.137024 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":47,"threads_started":1}
I20260926 05:46:04.138913 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 14521
I20260926 05:46:04.144253 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 14596
I20260926 05:46:04.149209 24407 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPYhTSt/build/release/bin/kudu with pid 14671
2026-09-26T05:46:04Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2781 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5562 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (271980 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST